Nov 03 20:28:07.076608 ip-10-0-138-26 systemd[1]: Starting Kubernetes Kubelet... Nov 03 20:28:07.509883 ip-10-0-138-26 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:28:07.509883 ip-10-0-138-26 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 20:28:07.509883 ip-10-0-138-26 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:28:07.510451 ip-10-0-138-26 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:28:07.515486 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515404 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:07.515486 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515483 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:07.515486 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515488 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:07.515486 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515492 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:07.515731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515496 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:07.515731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515501 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:07.515731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515505 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:07.515731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515509 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:07.515731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515513 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:07.515731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515524 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:07.515731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515528 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:07.515731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515532 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:07.515731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515536 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:07.515731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515540 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:07.515731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515544 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:07.515731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515548 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:07.515731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515552 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:07.515731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515557 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:07.515731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515561 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:07.515731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515565 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:07.515731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515570 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:07.515731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515574 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:07.515731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515578 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:07.516529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515582 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:07.516529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515586 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:07.516529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515590 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:07.516529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515594 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:07.516529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515599 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:07.516529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515603 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:07.516529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515608 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:07.516529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515612 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:07.516529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515616 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:07.516529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515621 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:07.516529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515625 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:07.516529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515629 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:07.516529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515634 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:07.516529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515641 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:07.516529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515651 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:07.516529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515657 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:07.516529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515662 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:07.516529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515667 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:07.516529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515672 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:07.517309 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515676 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:07.517309 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515680 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:07.517309 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515684 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:07.517309 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515689 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:07.517309 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515694 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:07.517309 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515698 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:07.517309 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515703 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:07.517309 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515707 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:07.517309 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515711 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:07.517309 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515715 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:07.517309 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515721 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:07.517309 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515725 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:07.517309 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515729 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:07.517309 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515733 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:07.517309 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515738 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:07.517309 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515742 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:07.517309 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515746 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:07.517309 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515752 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:07.517309 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515757 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:07.518125 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515761 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:07.518125 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515766 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:07.518125 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515771 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:07.518125 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515775 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:07.518125 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515779 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:07.518125 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515784 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:07.518125 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515788 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:07.518125 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515793 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:07.518125 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515797 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:07.518125 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515802 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:07.518125 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515807 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:07.518125 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515812 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:07.518125 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515816 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:07.518125 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515821 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:07.518125 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515825 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:07.518125 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515829 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:07.518125 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515833 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:07.518125 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515838 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:07.518125 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515842 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:07.518125 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515847 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:07.519013 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515851 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:07.519013 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515855 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:07.519013 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515859 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:07.519013 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515863 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:07.519013 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515867 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:07.519013 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515871 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:07.519013 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515876 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:07.519013 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515881 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:07.519013 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515886 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:07.519013 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515893 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:07.519013 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515899 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:07.519013 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515905 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:07.519013 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515909 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:07.519013 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515914 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:07.519013 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515918 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:07.519013 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515923 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:07.519013 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515927 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:07.519013 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515932 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:07.519013 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515938 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:07.519616 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515943 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:07.519616 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515947 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:07.519616 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515951 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:07.519616 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515957 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:07.519616 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515962 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:07.519616 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.515967 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:07.519616 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517309 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:07.519616 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517321 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:07.519616 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517326 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:07.519616 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517352 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:07.519616 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517357 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:07.519616 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517361 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:07.519616 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517366 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:07.519616 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517370 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:07.519616 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517375 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:07.519616 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517380 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:07.519616 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517385 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:07.519616 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517389 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:07.519616 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517393 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:07.520110 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517398 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:07.520110 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517402 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:07.520110 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517407 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:07.520110 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517412 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:07.520110 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517416 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:07.520110 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517421 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:07.520110 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517425 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:07.520110 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517430 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:07.520110 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517434 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:07.520110 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517441 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:07.520110 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517448 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:07.520110 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517452 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:07.520110 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517457 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:07.520110 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517462 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:07.520110 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517467 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:07.520110 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517471 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:07.520110 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517475 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:07.520110 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517481 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:07.520110 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517486 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:07.520765 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517491 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:07.520765 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517496 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:07.520765 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517501 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:07.520765 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517505 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:07.520765 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517509 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:07.520765 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517514 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:07.520765 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517518 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:07.520765 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517525 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:07.520765 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517533 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:07.520765 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517538 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:07.520765 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517543 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:07.520765 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517548 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:07.520765 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517553 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:07.520765 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517557 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:07.520765 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517562 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:07.520765 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517567 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:07.520765 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517572 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:07.520765 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517576 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:07.520765 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517581 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:07.520765 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517585 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:07.521533 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517589 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:07.521533 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517593 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:07.521533 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517598 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:07.521533 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517602 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:07.521533 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517607 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:07.521533 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517611 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:07.521533 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517615 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:07.521533 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517620 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:07.521533 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517624 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:07.521533 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517628 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:07.521533 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517633 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:07.521533 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517638 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:07.521533 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517643 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:07.521533 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517647 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:07.521533 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517651 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:07.521533 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517656 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:07.521533 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517660 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:07.521533 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517664 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:07.521533 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517668 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:07.521533 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517673 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:07.522101 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517677 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:07.522101 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517682 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:07.522101 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517686 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:07.522101 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517691 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:07.522101 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517695 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:07.522101 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517699 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:07.522101 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517703 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:07.522101 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517708 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:07.522101 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517713 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:07.522101 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517718 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:07.522101 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517722 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:07.522101 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517726 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:07.522101 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517730 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:07.522101 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517734 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:07.522101 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517739 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:07.522101 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517743 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:07.522101 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517747 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:07.522101 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517751 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:07.522101 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517756 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:07.522821 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517760 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:07.522821 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517764 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:07.522821 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517769 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:07.522821 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517774 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:07.522821 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517778 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:07.522821 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517783 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:07.522821 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517787 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:07.522821 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517793 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:07.522821 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517798 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:07.522821 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517802 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:07.522821 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517806 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:07.522821 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517810 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:07.522821 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517815 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:07.522821 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517819 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:07.522821 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.517823 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:07.522821 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518653 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:07.522821 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518662 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:07.522821 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518667 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:07.522821 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518671 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:07.522821 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518676 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:07.523311 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518681 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:07.523311 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518686 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:07.523311 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518690 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:07.523311 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518694 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:07.523311 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518698 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:07.523311 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518702 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:07.523311 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518707 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:07.523311 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518712 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:07.523311 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518716 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:07.523311 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518721 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:07.523311 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518726 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:07.523311 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518732 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:07.523311 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518736 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:07.523311 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518741 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:07.523311 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518745 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:07.523311 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518749 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:07.523311 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518753 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:07.523311 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518758 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:07.523311 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518763 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:07.523776 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518767 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:07.523776 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518771 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:07.523776 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518776 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:07.523776 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518780 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:07.523776 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518785 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:07.523776 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518789 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:07.523776 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518794 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:07.523776 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518798 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:07.523776 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518802 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:07.523776 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518806 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:07.523776 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518810 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:07.523776 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518815 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:07.523776 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518820 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:07.523776 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518824 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:07.523776 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518828 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:07.523776 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518833 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:07.523776 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518837 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:07.523776 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518842 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:07.523776 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518846 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:07.523776 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518851 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:07.524251 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518856 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:07.524251 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518860 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:07.524251 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518864 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:07.524251 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518869 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:07.524251 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518873 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:07.524251 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518877 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:07.524251 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518881 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:07.524251 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518885 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:07.524251 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518890 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:07.524251 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518894 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:07.524251 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518898 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:07.524251 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518903 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:07.524251 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518907 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:07.524251 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518912 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:07.524251 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518917 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:07.524251 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518921 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:07.524251 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518925 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:07.524251 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518929 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:07.524251 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518933 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:07.524251 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518938 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:07.524764 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518943 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:07.524764 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518947 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:07.524764 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518951 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:07.524764 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518956 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:07.524764 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518960 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:07.524764 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518967 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:07.524764 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518973 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:07.524764 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518979 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:07.524764 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518985 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:07.524764 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518991 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:07.524764 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.518995 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:07.524764 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519002 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:07.524764 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519006 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:07.524764 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519010 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:07.524764 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519014 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:07.524764 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519018 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:07.524764 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519022 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:07.524764 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519026 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:07.524764 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519030 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:07.525301 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519034 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:07.525301 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519038 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:07.525301 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519043 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:07.525301 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519047 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:07.525301 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519051 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:07.525301 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519057 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:07.525301 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519061 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:07.525301 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519065 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:07.525301 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519071 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:07.525301 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519076 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:07.525301 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519080 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:07.525301 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519084 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:07.525301 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519088 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:07.525301 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519093 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:07.525301 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519097 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:07.525301 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519102 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:07.525301 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519106 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:07.525301 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519111 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:07.525301 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519115 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:07.525775 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519119 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:07.525775 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519123 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:07.525775 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519127 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:07.525775 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.519131 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:07.525775 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520398 2572 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 20:28:07.525775 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520421 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 20:28:07.525775 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520431 2572 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 20:28:07.525775 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520440 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 20:28:07.525775 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520447 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 20:28:07.525775 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520452 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 20:28:07.525775 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520458 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 20:28:07.525775 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520465 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 20:28:07.525775 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520470 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 20:28:07.525775 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520475 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 20:28:07.525775 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520481 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 20:28:07.525775 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520486 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 20:28:07.525775 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520491 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 20:28:07.525775 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520497 2572 flags.go:64] FLAG: --cgroup-root="" Nov 03 20:28:07.525775 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520501 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 20:28:07.525775 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520507 2572 flags.go:64] FLAG: --client-ca-file="" Nov 03 20:28:07.525775 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520512 2572 flags.go:64] FLAG: --cloud-provider="external" Nov 03 20:28:07.525775 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520517 2572 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 20:28:07.525775 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520524 2572 flags.go:64] FLAG: --cluster-domain="" Nov 03 20:28:07.526311 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520529 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 20:28:07.526311 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520534 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 20:28:07.526311 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520539 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 20:28:07.526311 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520548 2572 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 20:28:07.526311 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520555 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 20:28:07.526311 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520560 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 20:28:07.526311 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520565 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 20:28:07.526311 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520570 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 20:28:07.526311 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520575 2572 flags.go:64] FLAG: --contention-profiling="false" Nov 03 20:28:07.526311 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520580 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 20:28:07.526311 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520584 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 20:28:07.526311 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520589 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 20:28:07.526311 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520594 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 20:28:07.526311 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520602 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 20:28:07.526311 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520608 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 20:28:07.526311 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520613 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 20:28:07.526311 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520618 2572 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 20:28:07.526311 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520623 2572 flags.go:64] FLAG: --enable-server="true" Nov 03 20:28:07.526311 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520628 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 20:28:07.526311 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520634 2572 flags.go:64] FLAG: --event-burst="100" Nov 03 20:28:07.526311 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520640 2572 flags.go:64] FLAG: --event-qps="50" Nov 03 20:28:07.526311 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520645 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 20:28:07.526311 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520649 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 20:28:07.526311 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520654 2572 flags.go:64] FLAG: --eviction-hard="" Nov 03 20:28:07.526949 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520661 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 20:28:07.526949 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520666 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 20:28:07.526949 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520670 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 20:28:07.526949 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520675 2572 flags.go:64] FLAG: --eviction-soft="" Nov 03 20:28:07.526949 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520680 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 20:28:07.526949 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520693 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 20:28:07.526949 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520698 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 20:28:07.526949 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520703 2572 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 20:28:07.526949 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520708 2572 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 20:28:07.526949 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520712 2572 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 20:28:07.526949 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520717 2572 flags.go:64] FLAG: --feature-gates="" Nov 03 20:28:07.526949 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520725 2572 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 20:28:07.526949 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520730 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 20:28:07.526949 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520735 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 20:28:07.526949 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520740 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 20:28:07.526949 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520745 2572 flags.go:64] FLAG: --healthz-port="10248" Nov 03 20:28:07.526949 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520750 2572 flags.go:64] FLAG: --help="false" Nov 03 20:28:07.526949 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520755 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:07.526949 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520760 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 20:28:07.526949 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520765 2572 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 20:28:07.526949 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520770 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 20:28:07.526949 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520775 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 20:28:07.526949 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520781 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 20:28:07.527504 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520785 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 20:28:07.527504 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520790 2572 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 20:28:07.527504 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520794 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 20:28:07.527504 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520799 2572 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 20:28:07.527504 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520804 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 20:28:07.527504 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520809 2572 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 20:28:07.527504 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520814 2572 flags.go:64] FLAG: --kube-reserved="" Nov 03 20:28:07.527504 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520819 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 20:28:07.527504 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520824 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 20:28:07.527504 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520829 2572 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 20:28:07.527504 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520834 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 20:28:07.527504 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520838 2572 flags.go:64] FLAG: --lock-file="" Nov 03 20:28:07.527504 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520843 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 20:28:07.527504 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520848 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 20:28:07.527504 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520853 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 20:28:07.527504 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520862 2572 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 20:28:07.527504 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520867 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 20:28:07.527504 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520872 2572 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 20:28:07.527504 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520876 2572 flags.go:64] FLAG: --logging-format="text" Nov 03 20:28:07.527504 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520881 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 20:28:07.527504 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520887 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 20:28:07.527504 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520892 2572 flags.go:64] FLAG: --manifest-url="" Nov 03 20:28:07.527504 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520896 2572 flags.go:64] FLAG: --manifest-url-header="" Nov 03 20:28:07.527504 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520903 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 20:28:07.527504 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520908 2572 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 20:28:07.528086 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520914 2572 flags.go:64] FLAG: --max-pods="110" Nov 03 20:28:07.528086 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520918 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 20:28:07.528086 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520923 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 20:28:07.528086 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520928 2572 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 20:28:07.528086 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520933 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 20:28:07.528086 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520938 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 20:28:07.528086 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520943 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 20:28:07.528086 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520947 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 20:28:07.528086 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520969 2572 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 20:28:07.528086 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520974 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 20:28:07.528086 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520979 2572 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 20:28:07.528086 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520984 2572 flags.go:64] FLAG: --pod-cidr="" Nov 03 20:28:07.528086 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520989 2572 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 20:28:07.528086 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520994 2572 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 20:28:07.528086 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.520998 2572 flags.go:64] FLAG: --pods-per-core="0" Nov 03 20:28:07.528086 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521003 2572 flags.go:64] FLAG: --port="10250" Nov 03 20:28:07.528086 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521007 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 20:28:07.528086 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521012 2572 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-049fc227c706e7617" Nov 03 20:28:07.528086 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521017 2572 flags.go:64] FLAG: --qos-reserved="" Nov 03 20:28:07.528086 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521022 2572 flags.go:64] FLAG: --read-only-port="10255" Nov 03 20:28:07.528086 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521027 2572 flags.go:64] FLAG: --register-node="true" Nov 03 20:28:07.528086 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521031 2572 flags.go:64] FLAG: --register-with-taints="" Nov 03 20:28:07.528086 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521043 2572 flags.go:64] FLAG: --registry-burst="10" Nov 03 20:28:07.528086 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521049 2572 flags.go:64] FLAG: --registry-qps="5" Nov 03 20:28:07.528086 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521054 2572 flags.go:64] FLAG: --reserved-cpus="" Nov 03 20:28:07.528086 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521059 2572 flags.go:64] FLAG: --reserved-memory="" Nov 03 20:28:07.528719 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521065 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 20:28:07.528719 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521070 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 20:28:07.528719 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521075 2572 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 20:28:07.528719 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521080 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 20:28:07.528719 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521085 2572 flags.go:64] FLAG: --runonce="false" Nov 03 20:28:07.528719 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521089 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 20:28:07.528719 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521094 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 20:28:07.528719 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521099 2572 flags.go:64] FLAG: --seccomp-default="false" Nov 03 20:28:07.528719 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521104 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 20:28:07.528719 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521109 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 20:28:07.528719 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521113 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 20:28:07.528719 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521119 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 20:28:07.528719 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521124 2572 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 20:28:07.528719 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521128 2572 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 20:28:07.528719 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521133 2572 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 20:28:07.528719 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521138 2572 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 20:28:07.528719 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521143 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 20:28:07.528719 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521148 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 20:28:07.528719 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521154 2572 flags.go:64] FLAG: --system-cgroups="" Nov 03 20:28:07.528719 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521159 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 20:28:07.528719 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521167 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 20:28:07.528719 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521172 2572 flags.go:64] FLAG: --tls-cert-file="" Nov 03 20:28:07.528719 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521180 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 20:28:07.528719 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521187 2572 flags.go:64] FLAG: --tls-min-version="" Nov 03 20:28:07.528719 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521192 2572 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 20:28:07.529321 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521196 2572 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 20:28:07.529321 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521201 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 20:28:07.529321 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521206 2572 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 20:28:07.529321 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521211 2572 flags.go:64] FLAG: --v="2" Nov 03 20:28:07.529321 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521218 2572 flags.go:64] FLAG: --version="false" Nov 03 20:28:07.529321 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521225 2572 flags.go:64] FLAG: --vmodule="" Nov 03 20:28:07.529321 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521231 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 20:28:07.529321 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521236 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 20:28:07.529321 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521428 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:07.529321 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521442 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:07.529321 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521446 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:07.529321 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521449 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:07.529321 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521452 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:07.529321 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521455 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:07.529321 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521458 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:07.529321 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521461 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:07.529321 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521464 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:07.529321 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521466 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:07.529321 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521469 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:07.529321 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521472 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:07.529321 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521474 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:07.529321 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521477 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:07.529888 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521480 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:07.529888 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521482 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:07.529888 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521485 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:07.529888 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521488 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:07.529888 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521491 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:07.529888 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521494 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:07.529888 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521497 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:07.529888 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521500 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:07.529888 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521503 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:07.529888 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521505 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:07.529888 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521509 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:07.529888 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521512 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:07.529888 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521514 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:07.529888 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521517 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:07.529888 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521520 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:07.529888 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521522 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:07.529888 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521525 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:07.529888 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521528 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:07.529888 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521530 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:07.529888 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521534 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:07.530388 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521537 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:07.530388 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521540 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:07.530388 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521543 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:07.530388 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521546 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:07.530388 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521549 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:07.530388 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521551 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:07.530388 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521554 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:07.530388 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521556 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:07.530388 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521559 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:07.530388 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521562 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:07.530388 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521564 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:07.530388 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521567 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:07.530388 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521570 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:07.530388 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521573 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:07.530388 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521575 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:07.530388 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521578 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:07.530388 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521580 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:07.530388 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521583 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:07.530388 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521585 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:07.530880 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521588 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:07.530880 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521591 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:07.530880 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521593 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:07.530880 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521596 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:07.530880 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521599 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:07.530880 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521602 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:07.530880 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521607 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:07.530880 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521612 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:07.530880 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521615 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:07.530880 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521618 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:07.530880 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521621 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:07.530880 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521624 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:07.530880 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521629 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:07.530880 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521632 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:07.530880 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521635 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:07.530880 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521637 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:07.530880 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521640 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:07.530880 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521643 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:07.530880 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521645 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:07.531327 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521648 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:07.531327 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521651 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:07.531327 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521654 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:07.531327 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521656 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:07.531327 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521659 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:07.531327 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521661 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:07.531327 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521665 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:07.531327 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521669 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:07.531327 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521671 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:07.531327 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521674 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:07.531327 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521677 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:07.531327 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521679 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:07.531327 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521682 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:07.531327 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521684 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:07.531327 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521687 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:07.531327 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521689 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:07.531327 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521692 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:07.531327 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521695 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:07.531327 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521699 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:07.531327 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521702 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:07.531850 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521705 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:07.531850 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521707 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:07.531850 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521709 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:07.531850 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521712 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:07.531850 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521715 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:07.531850 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521719 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:07.531850 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521721 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:07.531850 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521724 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:07.531850 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521727 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:07.531850 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521729 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:07.531850 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521732 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:07.531850 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521734 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:07.531850 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521737 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:07.531850 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.521739 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:07.531850 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.521769 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:28:07.531850 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.522985 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:07.532233 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523053 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:07.532233 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523174 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:07.532233 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523181 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:07.532233 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523186 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:07.532233 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523192 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:07.532233 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523197 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:07.532233 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523202 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:07.532233 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523206 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:07.532233 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523212 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:07.532233 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523216 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:07.532233 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523220 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:07.532233 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523225 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:07.532233 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523229 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:07.532233 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523232 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:07.532233 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523237 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:07.532233 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523241 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:07.532233 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523245 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:07.532233 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523251 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:07.532233 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523258 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:07.532233 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523264 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:07.532731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523269 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:07.532731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523273 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:07.532731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523277 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:07.532731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523281 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:07.532731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523285 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:07.532731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523289 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:07.532731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523293 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:07.532731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523298 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:07.532731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523302 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:07.532731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523306 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:07.532731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523310 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:07.532731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523314 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:07.532731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523319 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:07.532731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523327 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:07.532731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523349 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:07.532731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523354 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:07.532731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523359 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:07.532731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523366 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:07.532731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523370 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:07.532731 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523374 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:07.533206 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523379 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:07.533206 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523383 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:07.533206 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523387 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:07.533206 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523392 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:07.533206 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523398 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:07.533206 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523403 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:07.533206 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523407 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:07.533206 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523412 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:07.533206 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523415 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:07.533206 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523418 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:07.533206 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523420 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:07.533206 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523423 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:07.533206 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523426 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:07.533206 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523429 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:07.533206 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523431 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:07.533206 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523434 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:07.533206 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523436 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:07.533206 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523440 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:07.533206 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523445 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:07.533206 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523450 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:07.533690 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523454 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:07.533690 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523457 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:07.533690 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523460 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:07.533690 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523463 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:07.533690 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523467 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:07.533690 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523471 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:07.533690 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523474 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:07.533690 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523477 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:07.533690 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523480 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:07.533690 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523483 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:07.533690 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523485 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:07.533690 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523488 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:07.533690 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523491 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:07.533690 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523493 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:07.533690 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523498 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:07.533690 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523501 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:07.533690 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523503 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:07.533690 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523506 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:07.533690 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523509 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:07.533690 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523512 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:07.534171 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523514 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:07.534171 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523517 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:07.534171 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523520 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:07.534171 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523522 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:07.534171 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523525 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:07.534171 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523528 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:07.534171 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523531 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:07.534171 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523533 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:07.534171 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523536 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:07.534171 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523539 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:07.534171 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523541 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:07.534171 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523544 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:07.534171 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523546 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:07.534171 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523549 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:07.534171 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523552 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:07.534171 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523554 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:07.534171 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523558 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:07.534171 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523562 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:07.534171 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523564 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:07.534641 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523567 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:07.534641 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523569 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:07.534641 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523573 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:07.534641 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523575 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:07.534641 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523578 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:07.534641 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.523583 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:07.534641 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.531061 2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 20:28:07.534641 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.531075 2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 20:28:07.534641 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.531747 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 20:28:07.534641 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.531753 2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 20:28:07.534641 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531809 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:07.534641 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531815 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:07.534641 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531819 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:07.534641 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531822 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:07.534641 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531825 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:07.534641 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531828 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:07.534641 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531831 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:07.534641 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531834 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:07.534641 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531837 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:07.535092 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531839 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:07.535092 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531842 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:07.535092 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531845 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:07.535092 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531848 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:07.535092 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531851 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:07.535092 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531854 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:07.535092 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531857 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:07.535092 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531860 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:07.535092 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531863 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:07.535092 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531865 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:07.535092 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531868 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:07.535092 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531871 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:07.535092 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531874 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:07.535092 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531877 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:07.535092 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531880 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:07.535092 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531882 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:07.535092 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531885 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:07.535092 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531888 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:07.535092 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531890 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:07.535092 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531894 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:07.535752 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531897 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:07.535752 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531899 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:07.535752 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531902 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:07.535752 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531905 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:07.535752 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531907 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:07.535752 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531910 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:07.535752 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531912 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:07.535752 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531915 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:07.535752 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531917 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:07.535752 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531921 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:07.535752 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531924 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:07.535752 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531927 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:07.535752 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531929 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:07.535752 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531932 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:07.535752 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531934 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:07.535752 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531937 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:07.535752 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531939 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:07.535752 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531942 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:07.535752 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531945 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:07.535752 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531947 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:07.536255 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531950 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:07.536255 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531952 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:07.536255 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531955 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:07.536255 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531959 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:07.536255 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531963 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:07.536255 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531967 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:07.536255 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531969 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:07.536255 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531972 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:07.536255 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531976 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:07.536255 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531979 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:07.536255 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531982 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:07.536255 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531985 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:07.536255 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531989 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:07.536255 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531992 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:07.536255 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531994 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:07.536255 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.531997 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:07.536255 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532000 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:07.536255 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532002 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:07.536255 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532005 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:07.536754 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532008 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:07.536754 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532010 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:07.536754 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532013 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:07.536754 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532016 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:07.536754 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532019 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:07.536754 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532022 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:07.536754 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532024 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:07.536754 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532027 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:07.536754 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532029 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:07.536754 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532032 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:07.536754 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532034 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:07.536754 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532037 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:07.536754 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532040 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:07.536754 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532042 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:07.536754 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532045 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:07.536754 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532048 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:07.536754 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532050 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:07.536754 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532054 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:07.536754 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532057 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:07.537193 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532059 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:07.537193 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532062 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:07.537193 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532065 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:07.537193 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532067 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:07.537193 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532070 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:07.537193 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532073 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:07.537193 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532076 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:07.537193 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532079 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:07.537193 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532081 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:07.537193 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532084 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:07.537193 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532087 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:07.537193 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532090 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:07.537193 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532092 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:07.537193 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532095 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:07.537193 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532098 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:07.537193 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532100 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:07.537193 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532103 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:07.537193 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532106 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:07.537193 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532109 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:07.537667 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.532138 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:28:07.537667 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532276 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:07.537667 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532281 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:07.537667 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532284 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:07.537667 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532288 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:07.537667 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532291 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:07.537667 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532293 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:07.537667 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532296 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:07.537667 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532299 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:07.537667 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532301 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:07.537667 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532304 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:07.537667 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532306 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:07.537667 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532309 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:07.537667 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532311 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:07.537667 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532314 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:07.537667 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532317 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:07.538049 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532319 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:07.538049 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532322 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:07.538049 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532325 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:07.538049 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532328 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:07.538049 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532351 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:07.538049 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532356 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:07.538049 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532359 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:07.538049 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532363 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:07.538049 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532366 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:07.538049 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532368 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:07.538049 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532371 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:07.538049 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532374 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:07.538049 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532376 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:07.538049 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532379 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:07.538049 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532382 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:07.538049 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532385 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:07.538049 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532388 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:07.538049 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532391 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:07.538049 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532393 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:07.538529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532396 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:07.538529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532398 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:07.538529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532401 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:07.538529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532404 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:07.538529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532406 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:07.538529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532409 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:07.538529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532411 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:07.538529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532414 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:07.538529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532417 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:07.538529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532419 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:07.538529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532422 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:07.538529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532424 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:07.538529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532428 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:07.538529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532432 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:07.538529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532437 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:07.538529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532441 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:07.538529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532446 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:07.538529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532450 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:07.538529 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532454 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:07.539021 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532457 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:07.539021 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532459 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:07.539021 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532462 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:07.539021 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532464 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:07.539021 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532467 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:07.539021 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532469 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:07.539021 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532472 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:07.539021 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532475 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:07.539021 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532477 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:07.539021 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532480 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:07.539021 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532483 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:07.539021 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532486 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:07.539021 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532489 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:07.539021 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532491 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:07.539021 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532494 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:07.539021 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532496 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:07.539021 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532499 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:07.539021 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532502 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:07.539021 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532504 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:07.539021 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532506 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:07.539633 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532509 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:07.539633 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532512 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:07.539633 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532514 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:07.539633 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532517 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:07.539633 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532520 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:07.539633 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532523 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:07.539633 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532525 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:07.539633 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532528 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:07.539633 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532531 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:07.539633 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532534 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:07.539633 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532536 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:07.539633 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532539 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:07.539633 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532541 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:07.539633 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532544 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:07.539633 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532546 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:07.539633 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532549 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:07.539633 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532552 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:07.539633 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532554 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:07.539633 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532557 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:07.539633 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532559 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:07.540122 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532563 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:07.540122 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532567 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:07.540122 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532571 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:07.540122 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532574 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:07.540122 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532578 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:07.540122 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532581 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:07.540122 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532583 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:07.540122 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532585 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:07.540122 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532588 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:07.540122 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532591 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:07.540122 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532593 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:07.540122 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532595 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:07.540122 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532598 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:07.540122 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.532623 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:28:07.540122 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532673 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:07.540605 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532678 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:07.540605 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532681 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:07.540605 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532692 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:07.540605 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532695 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:07.540605 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532698 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:07.540605 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532701 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:07.540605 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532704 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:07.540605 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532706 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:07.540605 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532709 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:07.540605 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532712 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:07.540605 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532714 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:07.540605 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532717 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:07.540605 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532719 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:07.540605 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532722 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:07.540605 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532724 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:07.540605 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532727 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:07.540605 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532729 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:07.540605 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532732 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:07.540605 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532735 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:07.541164 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532738 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:07.541164 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532740 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:07.541164 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532744 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:07.541164 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532747 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:07.541164 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532749 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:07.541164 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532752 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:07.541164 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532755 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:07.541164 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532757 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:07.541164 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532760 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:07.541164 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532762 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:07.541164 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532765 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:07.541164 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532767 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:07.541164 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532770 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:07.541164 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532772 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:07.541164 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532775 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:07.541164 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532777 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:07.541164 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532780 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:07.541164 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532783 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:07.541164 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532788 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:07.541164 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532791 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:07.541652 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532793 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:07.541652 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532796 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:07.541652 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532799 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:07.541652 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532802 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:07.541652 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532804 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:07.541652 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532806 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:07.541652 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532809 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:07.541652 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532811 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:07.541652 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532814 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:07.541652 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532817 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:07.541652 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532819 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:07.541652 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532822 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:07.541652 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532824 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:07.541652 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532827 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:07.541652 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532829 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:07.541652 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532834 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:07.541652 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532840 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:07.541652 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532844 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:07.541652 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532846 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:07.541652 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532849 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:07.542129 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532852 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:07.542129 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532854 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:07.542129 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532857 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:07.542129 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532860 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:07.542129 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532862 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:07.542129 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532865 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:07.542129 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532867 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:07.542129 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532870 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:07.542129 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532873 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:07.542129 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532875 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:07.542129 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532879 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:07.542129 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532882 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:07.542129 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532885 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:07.542129 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532887 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:07.542129 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532890 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:07.542129 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532893 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:07.542129 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532895 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:07.542129 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532898 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:07.542129 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532900 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:07.542639 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532902 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:07.542639 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532905 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:07.542639 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532908 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:07.542639 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532910 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:07.542639 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532913 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:07.542639 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532915 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:07.542639 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532918 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:07.542639 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532920 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:07.542639 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532924 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:07.542639 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532929 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:07.542639 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532933 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:07.542639 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532936 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:07.542639 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532938 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:07.542639 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532941 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:07.542639 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532943 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:07.542639 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532946 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:07.542639 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532948 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:07.542639 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532951 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:07.542639 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532953 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:07.543080 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532958 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:07.543080 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532960 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:07.543080 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532963 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:07.543080 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532965 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:07.543080 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532969 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:07.543080 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532972 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:07.543080 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532975 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:07.543080 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:07.532977 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:07.543080 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.533029 2572 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 20:28:07.543080 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.535432 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 20:28:07.543080 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.536442 2572 server.go:1014] "Starting client certificate rotation" Nov 03 20:28:07.543080 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.536543 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:28:07.543080 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.536579 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:28:07.543080 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.538913 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:28:07.543080 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.540127 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:28:07.552078 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.552061 2572 log.go:25] "Validated CRI v1 runtime API" Nov 03 20:28:07.555519 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.555504 2572 log.go:25] "Validated CRI v1 image API" Nov 03 20:28:07.556306 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.556276 2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 20:28:07.560766 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.560744 2572 fs.go:135] Filesystem UUIDs: map[2b4c1d62-c9a2-48f5-91ed-b06e6eb81532:/dev/nvme0n1p4 6ff11b39-2cad-4870-9d97-2394c1f6a523:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 03 20:28:07.560858 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.560767 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 20:28:07.566693 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.566577 2572 manager.go:217] Machine: {Timestamp:2026-11-03 20:28:07.566071066 +0000 UTC m=+0.363170607 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108583 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26fb0d2e4b72d42008dd6094eb0d31 SystemUUID:ec26fb0d-2e4b-72d4-2008-dd6094eb0d31 BootID:18484d08-debf-4fda-8c05-b01468453a28 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:c1:3c:25:81 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:c1:3c:25:81 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:df:55:fb:e3:e1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 20:28:07.567368 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.567356 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 20:28:07.567461 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.567449 2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 20:28:07.571530 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.571496 2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 20:28:07.571661 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.571532 2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-26.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 20:28:07.571702 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.571664 2572 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 20:28:07.571702 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.571671 2572 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 20:28:07.571750 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.571708 2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:28:07.571750 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.571727 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:28:07.573800 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.573790 2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 20:28:07.578201 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.578189 2572 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 20:28:07.578312 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.578303 2572 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 20:28:07.579172 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.579163 2572 kubelet.go:493] "Attempting to sync node with API server" Nov 03 20:28:07.579203 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.579177 2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 20:28:07.579203 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.579188 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 20:28:07.579203 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.579197 2572 kubelet.go:405] "Adding apiserver pod source" Nov 03 20:28:07.579289 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.579206 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 20:28:07.579594 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.579577 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:28:07.581888 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.581870 2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 20:28:07.587972 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.587959 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:28:07.588982 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.588969 2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 20:28:07.589025 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.589001 2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 20:28:07.589162 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.589151 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 20:28:07.589190 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.589169 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 20:28:07.589190 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.589175 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 20:28:07.589190 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.589181 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 20:28:07.589190 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.589186 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 20:28:07.589280 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.589192 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 20:28:07.590094 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.590086 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 20:28:07.590127 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.590095 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 20:28:07.590127 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.590103 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 20:28:07.590127 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.590116 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 20:28:07.590197 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.590128 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 20:28:07.590197 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.590137 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 20:28:07.592008 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.591997 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 20:28:07.592042 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.592011 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 20:28:07.595939 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.595927 2572 server.go:1263] "Started kubelet" Nov 03 20:28:07.596614 ip-10-0-138-26 systemd[1]: Started Kubernetes Kubelet. Nov 03 20:28:07.596702 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.596600 2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 20:28:07.597691 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.597641 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 20:28:07.597769 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.597707 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 20:28:07.598946 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.598925 2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 20:28:07.599954 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.599940 2572 server.go:317] "Adding debug handlers to kubelet server" Nov 03 20:28:07.604888 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:07.604868 2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 20:28:07.605942 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.605927 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 20:28:07.606573 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.606555 2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 20:28:07.606696 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:07.606664 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-26.us-west-2.compute.internal\" not found" Nov 03 20:28:07.607268 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.607248 2572 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 20:28:07.607268 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.607267 2572 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 20:28:07.607422 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.607248 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 20:28:07.607422 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.607308 2572 factory.go:55] Registering systemd factory Nov 03 20:28:07.607422 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.607361 2572 factory.go:223] Registration of the systemd container factory successfully Nov 03 20:28:07.607558 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.607512 2572 reconstruct.go:97] "Volume reconstruction finished" Nov 03 20:28:07.607558 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.607553 2572 reconciler.go:29] "Reconciler: start to sync state" Nov 03 20:28:07.607697 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.607578 2572 factory.go:153] Registering CRI-O factory Nov 03 20:28:07.607697 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.607591 2572 factory.go:223] Registration of the crio container factory successfully Nov 03 20:28:07.607697 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.607636 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 20:28:07.607697 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.607650 2572 factory.go:103] Registering Raw factory Nov 03 20:28:07.607697 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.607660 2572 manager.go:1199] Started watching for new ooms in manager Nov 03 20:28:07.608680 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.608666 2572 manager.go:319] Starting recovery of all containers Nov 03 20:28:07.620278 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.620257 2572 manager.go:324] Recovery completed Nov 03 20:28:07.624047 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.624034 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:07.626035 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.626013 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:07.626074 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.626054 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:07.626074 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.626067 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:07.626567 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.626554 2572 cpu_manager.go:226] "Starting" policy="none" Nov 03 20:28:07.626567 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.626563 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:07.626640 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.626566 2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 20:28:07.626640 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.626600 2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 20:28:07.627244 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.627224 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:07.627280 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.627251 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:07.627280 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.627262 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:07.628859 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.628844 2572 policy_none.go:50] "Start" Nov 03 20:28:07.628859 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.628861 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 20:28:07.628947 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.628871 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 20:28:07.630851 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.630839 2572 policy_none.go:44] "Start" Nov 03 20:28:07.681889 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.681854 2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 20:28:07.684367 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.684355 2572 manager.go:341] "Starting Device Plugin manager" Nov 03 20:28:07.684456 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:07.684384 2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 20:28:07.684456 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.684396 2572 server.go:87] "Starting device plugin registration server" Nov 03 20:28:07.684643 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.684629 2572 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 20:28:07.684710 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.684643 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 20:28:07.684761 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.684743 2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 20:28:07.684828 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.684815 2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 20:28:07.684828 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.684828 2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 20:28:07.685389 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:07.685293 2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 20:28:07.685389 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:07.685359 2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-26.us-west-2.compute.internal\" not found" Nov 03 20:28:07.728514 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:07.728484 2572 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-26.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 20:28:07.728652 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.728521 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-26.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 20:28:07.729597 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:07.728495 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-26.us-west-2.compute.internal.18e4a2ec5962785e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-26.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-26.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 20:28:07.595907166 +0000 UTC m=+0.393006706,LastTimestamp:2026-11-03 20:28:07.595907166 +0000 UTC m=+0.393006706,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-26.us-west-2.compute.internal,}" Nov 03 20:28:07.797785 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:07.784351 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-26.us-west-2.compute.internal.18e4a2ec5b2e36c4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-26.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-26.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-26.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 20:28:07.626036932 +0000 UTC m=+0.423136473,LastTimestamp:2026-11-03 20:28:07.626036932 +0000 UTC m=+0.423136473,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-26.us-west-2.compute.internal,}" Nov 03 20:28:07.797785 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.784700 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:07.797785 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:07.785298 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 20:28:07.797785 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:07.785292 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-26.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 20:28:07.797785 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:07.785298 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 20:28:07.797785 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.785546 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:07.797785 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.785569 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:07.797785 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.785580 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:07.797785 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.785605 2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:07.810830 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.810795 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 20:28:07.810911 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.810861 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xs4nb" Nov 03 20:28:07.829410 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.829377 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 20:28:07.830661 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.830641 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 20:28:07.830661 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.830662 2572 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 20:28:07.830764 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.830679 2572 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 20:28:07.830764 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:07.830714 2572 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 20:28:07.883225 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.883201 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:07.885477 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.885460 2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:07.885544 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:07.885485 2572 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-26.us-west-2.compute.internal\": node \"ip-10-0-138-26.us-west-2.compute.internal\" not found" Nov 03 20:28:07.931441 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.931417 2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal"] Nov 03 20:28:07.931516 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.931469 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:07.932356 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.932327 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:07.932422 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.932372 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:07.932422 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.932383 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:07.934597 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.934585 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:07.934746 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.934730 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:07.935310 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.935016 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:07.936876 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.936834 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:07.936961 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.936889 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:07.936961 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.936906 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:07.937054 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.936959 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:07.937054 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.936994 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:07.937054 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.937012 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:07.939456 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.939441 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:07.939527 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.939468 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:07.940261 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.940246 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:07.940323 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.940273 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:07.940323 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:07.940285 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:07.958645 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:07.958628 2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-26.us-west-2.compute.internal\" not found" node="ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:07.960584 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:07.960568 2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-26.us-west-2.compute.internal\" not found" node="ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:08.001020 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:08.000995 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-26.us-west-2.compute.internal\" not found" Nov 03 20:28:08.009282 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:08.009258 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cb4d50458eb702ff22ca2222c359eed8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal\" (UID: \"cb4d50458eb702ff22ca2222c359eed8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:08.009385 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:08.009284 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb4d50458eb702ff22ca2222c359eed8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal\" (UID: \"cb4d50458eb702ff22ca2222c359eed8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:08.009385 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:08.009302 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/733c8a62db18973abc6d8fb96bb89db1-config\") pod \"kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal\" (UID: \"733c8a62db18973abc6d8fb96bb89db1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:08.101678 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:08.101609 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-26.us-west-2.compute.internal\" not found" Nov 03 20:28:08.109955 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:08.109931 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cb4d50458eb702ff22ca2222c359eed8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal\" (UID: \"cb4d50458eb702ff22ca2222c359eed8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:08.110030 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:08.109960 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb4d50458eb702ff22ca2222c359eed8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal\" (UID: \"cb4d50458eb702ff22ca2222c359eed8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:08.110030 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:08.109977 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/733c8a62db18973abc6d8fb96bb89db1-config\") pod \"kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal\" (UID: \"733c8a62db18973abc6d8fb96bb89db1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:08.110116 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:08.110029 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/733c8a62db18973abc6d8fb96bb89db1-config\") pod \"kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal\" (UID: \"733c8a62db18973abc6d8fb96bb89db1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:08.110116 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:08.110045 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb4d50458eb702ff22ca2222c359eed8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal\" (UID: \"cb4d50458eb702ff22ca2222c359eed8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:08.110116 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:08.110047 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cb4d50458eb702ff22ca2222c359eed8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal\" (UID: \"cb4d50458eb702ff22ca2222c359eed8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:08.202231 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:08.202210 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-26.us-west-2.compute.internal\" not found" Nov 03 20:28:08.263375 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:08.263348 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:08.265519 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:08.265502 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:08.303346 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:08.303305 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-26.us-west-2.compute.internal\" not found" Nov 03 20:28:08.404183 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:08.404111 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-26.us-west-2.compute.internal\" not found" Nov 03 20:28:08.504664 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:08.504633 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-26.us-west-2.compute.internal\" not found" Nov 03 20:28:08.536004 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:08.535985 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 20:28:08.536412 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:08.536092 2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 20:28:08.536412 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:08.536093 2572 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a6eb617b3f8214fa495946204f12b38d-1c21b587483bc72e.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-138-26.us-west-2.compute.internal.18e4a2ec5b2ebd77\": read tcp 10.0.138.26:42068->3.149.15.198:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-138-26.us-west-2.compute.internal.18e4a2ec5b2ebd77 default 5030 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-26.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-26.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-26.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 20:28:07 +0000 UTC,LastTimestamp:2026-11-03 20:28:07.932387186 +0000 UTC m=+0.729486727,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-26.us-west-2.compute.internal,}" Nov 03 20:28:08.605509 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:08.605475 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-26.us-west-2.compute.internal\" not found" Nov 03 20:28:08.706144 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:08.706069 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-26.us-west-2.compute.internal\" not found" Nov 03 20:28:08.806294 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:08.806265 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-26.us-west-2.compute.internal\" not found" Nov 03 20:28:08.815635 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:08.815606 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 20:23:08 +0000 UTC" deadline="2028-03-29 19:06:56.281613422 +0000 UTC" Nov 03 20:28:08.815635 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:08.815633 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12286h38m47.46598283s" Nov 03 20:28:08.904366 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:08.904323 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:08.905320 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:08.905304 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:08.905444 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:08.905355 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:08.905444 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:08.905372 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:08.906393 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:08.906378 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-26.us-west-2.compute.internal\" not found" Nov 03 20:28:08.994033 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:08.993968 2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:09.006512 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:09.006487 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-26.us-west-2.compute.internal\" not found" Nov 03 20:28:09.079133 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.079103 2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:09.107186 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.107157 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:09.178992 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.178964 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:28:09.179191 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.179133 2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:09.179919 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.179904 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:09.242271 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.242248 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:28:09.508745 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:09.508712 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733c8a62db18973abc6d8fb96bb89db1.slice/crio-0c5b6a1b1f30d3e9122ef65b5a92319eaecd4674ea7acfb1194edbbb5ea43818 WatchSource:0}: Error finding container 0c5b6a1b1f30d3e9122ef65b5a92319eaecd4674ea7acfb1194edbbb5ea43818: Status 404 returned error can't find the container with id 0c5b6a1b1f30d3e9122ef65b5a92319eaecd4674ea7acfb1194edbbb5ea43818 Nov 03 20:28:09.509211 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:09.509197 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb4d50458eb702ff22ca2222c359eed8.slice/crio-b0742edfa33aa6021b9dc3a08e45c023e436ce141fd7b15572b3831f1a5a7c66 WatchSource:0}: Error finding container b0742edfa33aa6021b9dc3a08e45c023e436ce141fd7b15572b3831f1a5a7c66: Status 404 returned error can't find the container with id b0742edfa33aa6021b9dc3a08e45c023e436ce141fd7b15572b3831f1a5a7c66 Nov 03 20:28:09.514050 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.514034 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:28:09.583150 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.583127 2572 apiserver.go:52] "Watching apiserver" Nov 03 20:28:09.607437 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.607406 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 20:28:09.697043 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.696970 2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 20:28:09.697314 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.697296 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-kzsn2","openshift-multus/multus-additional-cni-plugins-tl7xc","openshift-multus/network-metrics-daemon-f28pp","kube-system/global-pull-secret-syncer-2sjv9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26","openshift-cluster-node-tuning-operator/tuned-bv4jb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-82vml","kube-system/konnectivity-agent-ccxwr","kube-system/kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal","openshift-image-registry/node-ca-5tx7t","openshift-multus/multus-pv8vt","openshift-network-diagnostics/network-check-target-l4pbv","openshift-network-operator/iptables-alerter-bjbkk"] Nov 03 20:28:09.701441 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.701422 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:09.701528 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:09.701509 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4pbv" podUID="6cd2ab07-4196-4ed1-b357-38f01b99b71d" Nov 03 20:28:09.703703 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.703685 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:09.705966 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.705938 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:09.706088 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:09.706019 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f28pp" podUID="2281c3da-747c-4a37-89ec-b764d8a95405" Nov 03 20:28:09.708275 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.708255 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:09.708369 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:09.708311 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sjv9" podUID="ef30d813-b7ed-417d-a9fe-85118bf77a95" Nov 03 20:28:09.710663 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.710641 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:09.713128 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.712987 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.716025 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.716005 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kzsn2" Nov 03 20:28:09.717439 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.717412 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nlpn\" (UniqueName: \"kubernetes.io/projected/6cd2ab07-4196-4ed1-b357-38f01b99b71d-kube-api-access-6nlpn\") pod \"network-check-target-l4pbv\" (UID: \"6cd2ab07-4196-4ed1-b357-38f01b99b71d\") " pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:09.717575 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.717445 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9844c086-efc5-46f3-9d37-34c2fa013f34-system-cni-dir\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:09.717575 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.717463 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9844c086-efc5-46f3-9d37-34c2fa013f34-cnibin\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:09.717575 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.717485 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9844c086-efc5-46f3-9d37-34c2fa013f34-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:09.717575 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.717536 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9487d\" (UniqueName: \"kubernetes.io/projected/2281c3da-747c-4a37-89ec-b764d8a95405-kube-api-access-9487d\") pod \"network-metrics-daemon-f28pp\" (UID: \"2281c3da-747c-4a37-89ec-b764d8a95405\") " pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:09.717575 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.717574 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef30d813-b7ed-417d-a9fe-85118bf77a95-original-pull-secret\") pod \"global-pull-secret-syncer-2sjv9\" (UID: \"ef30d813-b7ed-417d-a9fe-85118bf77a95\") " pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:09.717861 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.717604 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrs5m\" (UniqueName: \"kubernetes.io/projected/9844c086-efc5-46f3-9d37-34c2fa013f34-kube-api-access-zrs5m\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:09.717861 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.717658 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs\") pod \"network-metrics-daemon-f28pp\" (UID: \"2281c3da-747c-4a37-89ec-b764d8a95405\") " pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:09.717861 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.717710 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef30d813-b7ed-417d-a9fe-85118bf77a95-dbus\") pod \"global-pull-secret-syncer-2sjv9\" (UID: \"ef30d813-b7ed-417d-a9fe-85118bf77a95\") " pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:09.717861 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.717749 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9844c086-efc5-46f3-9d37-34c2fa013f34-cni-binary-copy\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:09.717861 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.717788 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9844c086-efc5-46f3-9d37-34c2fa013f34-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:09.717861 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.717824 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9844c086-efc5-46f3-9d37-34c2fa013f34-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:09.717861 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.717847 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef30d813-b7ed-417d-a9fe-85118bf77a95-kubelet-config\") pod \"global-pull-secret-syncer-2sjv9\" (UID: \"ef30d813-b7ed-417d-a9fe-85118bf77a95\") " pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:09.719714 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.719690 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.722216 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.722196 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ccxwr" Nov 03 20:28:09.724681 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.724665 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bjbkk" Nov 03 20:28:09.727318 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.727293 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5tx7t" Nov 03 20:28:09.729834 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.729818 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.761341 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.761320 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 20:28:09.761457 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.761390 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 20:28:09.761524 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.761514 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 20:28:09.761561 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.761528 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 20:28:09.761598 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.761562 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pxhx8\"" Nov 03 20:28:09.761598 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.761570 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 20:28:09.769862 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.769843 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 20:28:09.770037 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.770022 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w6582\"" Nov 03 20:28:09.770105 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.770065 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 20:28:09.770105 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.770086 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 20:28:09.770535 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.770521 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:28:09.770616 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.770604 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 20:28:09.770677 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.770654 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8pktm\"" Nov 03 20:28:09.808484 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.808457 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 20:28:09.809275 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.809262 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:09.818070 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818044 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9844c086-efc5-46f3-9d37-34c2fa013f34-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:09.818124 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818080 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-host\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.818124 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818097 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f7ab819a-4321-4958-a5d5-230cbdbd9428-serviceca\") pod \"node-ca-5tx7t\" (UID: \"f7ab819a-4321-4958-a5d5-230cbdbd9428\") " pod="openshift-image-registry/node-ca-5tx7t" Nov 03 20:28:09.818124 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818112 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-run-systemd\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.818223 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818129 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-run-netns\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.818223 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818195 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5125fef-6a2d-45d6-99d2-90915b575f32-multus-daemon-config\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.818296 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818286 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9844c086-efc5-46f3-9d37-34c2fa013f34-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:09.818336 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818313 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-sysctl-d\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.818394 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818338 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7ab819a-4321-4958-a5d5-230cbdbd9428-host\") pod \"node-ca-5tx7t\" (UID: \"f7ab819a-4321-4958-a5d5-230cbdbd9428\") " pod="openshift-image-registry/node-ca-5tx7t" Nov 03 20:28:09.818440 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818405 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-socket-dir\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:09.818440 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818431 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-registration-dir\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:09.818516 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818468 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-systemd-units\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.818516 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818500 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9844c086-efc5-46f3-9d37-34c2fa013f34-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:09.818581 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818512 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-ovn-node-metrics-cert\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.818581 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818548 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-system-cni-dir\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.818654 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818582 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9487d\" (UniqueName: \"kubernetes.io/projected/2281c3da-747c-4a37-89ec-b764d8a95405-kube-api-access-9487d\") pod \"network-metrics-daemon-f28pp\" (UID: \"2281c3da-747c-4a37-89ec-b764d8a95405\") " pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:09.818654 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818632 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4cc66ac6-6311-4144-8db1-3c39182e1a19-iptables-alerter-script\") pod \"iptables-alerter-bjbkk\" (UID: \"4cc66ac6-6311-4144-8db1-3c39182e1a19\") " pod="openshift-network-operator/iptables-alerter-bjbkk" Nov 03 20:28:09.818741 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818727 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-kubelet\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.818784 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818738 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9844c086-efc5-46f3-9d37-34c2fa013f34-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:09.818784 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818752 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.818784 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818771 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-var-lib-cni-bin\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.818889 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818802 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-multus-conf-dir\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.818889 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818832 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrlqh\" (UniqueName: \"kubernetes.io/projected/d5125fef-6a2d-45d6-99d2-90915b575f32-kube-api-access-hrlqh\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.818889 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818853 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-run\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.818889 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818878 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-etc-openvswitch\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.819021 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818905 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-run-k8s-cni-cncf-io\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.819021 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818938 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-var-lib-cni-multus\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.819021 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818963 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e99c666f-7bb3-4856-a2fc-935ee184dae3-konnectivity-ca\") pod \"konnectivity-agent-ccxwr\" (UID: \"e99c666f-7bb3-4856-a2fc-935ee184dae3\") " pod="kube-system/konnectivity-agent-ccxwr" Nov 03 20:28:09.819021 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.818990 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-cni-bin\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.819203 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819031 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef30d813-b7ed-417d-a9fe-85118bf77a95-kubelet-config\") pod \"global-pull-secret-syncer-2sjv9\" (UID: \"ef30d813-b7ed-417d-a9fe-85118bf77a95\") " pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:09.819203 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819071 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef30d813-b7ed-417d-a9fe-85118bf77a95-kubelet-config\") pod \"global-pull-secret-syncer-2sjv9\" (UID: \"ef30d813-b7ed-417d-a9fe-85118bf77a95\") " pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:09.819203 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819131 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59643abf-923a-4d4d-8235-c2ad22c8aea0-tmp-dir\") pod \"node-resolver-kzsn2\" (UID: \"59643abf-923a-4d4d-8235-c2ad22c8aea0\") " pod="openshift-dns/node-resolver-kzsn2" Nov 03 20:28:09.819203 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819174 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-run-openvswitch\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.819433 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819206 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-ovnkube-config\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.819433 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819241 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5125fef-6a2d-45d6-99d2-90915b575f32-cni-binary-copy\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.819433 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819271 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-var-lib-kubelet\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.819433 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819301 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9844c086-efc5-46f3-9d37-34c2fa013f34-system-cni-dir\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:09.819433 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819326 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9844c086-efc5-46f3-9d37-34c2fa013f34-cnibin\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:09.819433 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819373 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9844c086-efc5-46f3-9d37-34c2fa013f34-cnibin\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:09.819433 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819376 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9844c086-efc5-46f3-9d37-34c2fa013f34-system-cni-dir\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:09.819433 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819374 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-etc-selinux\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:09.819433 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819417 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-run-ovn\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.819811 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819454 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-run-ovn-kubernetes\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.819811 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819483 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs\") pod \"network-metrics-daemon-f28pp\" (UID: \"2281c3da-747c-4a37-89ec-b764d8a95405\") " pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:09.819811 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819510 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdhpf\" (UniqueName: \"kubernetes.io/projected/4cc66ac6-6311-4144-8db1-3c39182e1a19-kube-api-access-vdhpf\") pod \"iptables-alerter-bjbkk\" (UID: \"4cc66ac6-6311-4144-8db1-3c39182e1a19\") " pod="openshift-network-operator/iptables-alerter-bjbkk" Nov 03 20:28:09.819811 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819537 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:09.819811 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819556 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fhqp\" (UniqueName: \"kubernetes.io/projected/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-kube-api-access-4fhqp\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:09.819811 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819593 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-run-multus-certs\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.819811 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:09.819600 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:09.819811 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819623 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9844c086-efc5-46f3-9d37-34c2fa013f34-cni-binary-copy\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:09.819811 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819647 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-systemd\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.819811 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:09.819677 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs podName:2281c3da-747c-4a37-89ec-b764d8a95405 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:10.319656721 +0000 UTC m=+3.116756270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs") pod "network-metrics-daemon-f28pp" (UID: "2281c3da-747c-4a37-89ec-b764d8a95405") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:09.819811 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819699 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-tuned\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.819811 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819733 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e99c666f-7bb3-4856-a2fc-935ee184dae3-agent-certs\") pod \"konnectivity-agent-ccxwr\" (UID: \"e99c666f-7bb3-4856-a2fc-935ee184dae3\") " pod="kube-system/konnectivity-agent-ccxwr" Nov 03 20:28:09.819811 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819765 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-multus-cni-dir\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.819811 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819789 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-var-lib-kubelet\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.819811 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819815 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-hostroot\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.820579 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819848 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-sysconfig\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.820579 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819914 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nlpn\" (UniqueName: \"kubernetes.io/projected/6cd2ab07-4196-4ed1-b357-38f01b99b71d-kube-api-access-6nlpn\") pod \"network-check-target-l4pbv\" (UID: \"6cd2ab07-4196-4ed1-b357-38f01b99b71d\") " pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:09.820579 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.819945 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9844c086-efc5-46f3-9d37-34c2fa013f34-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:09.820579 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820022 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef30d813-b7ed-417d-a9fe-85118bf77a95-original-pull-secret\") pod \"global-pull-secret-syncer-2sjv9\" (UID: \"ef30d813-b7ed-417d-a9fe-85118bf77a95\") " pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:09.820579 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820084 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9844c086-efc5-46f3-9d37-34c2fa013f34-cni-binary-copy\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:09.820579 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820096 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-lib-modules\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.820579 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:09.820168 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:09.820579 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820172 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-multus-socket-dir-parent\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.820579 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:09.820233 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef30d813-b7ed-417d-a9fe-85118bf77a95-original-pull-secret podName:ef30d813-b7ed-417d-a9fe-85118bf77a95 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:10.320220708 +0000 UTC m=+3.117320238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef30d813-b7ed-417d-a9fe-85118bf77a95-original-pull-secret") pod "global-pull-secret-syncer-2sjv9" (UID: "ef30d813-b7ed-417d-a9fe-85118bf77a95") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:09.820579 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820276 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-modprobe-d\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.820579 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820300 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-kubernetes\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.820579 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820323 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5474343-621e-4dd9-948c-ad71bd6f00c2-tmp\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.820579 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820368 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-ovnkube-script-lib\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.820579 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820391 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-etc-kubernetes\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.820579 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820436 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cc66ac6-6311-4144-8db1-3c39182e1a19-host-slash\") pod \"iptables-alerter-bjbkk\" (UID: \"4cc66ac6-6311-4144-8db1-3c39182e1a19\") " pod="openshift-network-operator/iptables-alerter-bjbkk" Nov 03 20:28:09.820579 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820485 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-device-dir\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:09.821277 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820499 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9844c086-efc5-46f3-9d37-34c2fa013f34-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:09.821277 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820517 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-run-netns\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.821277 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820600 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-env-overrides\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.821277 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820625 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s62xs\" (UniqueName: \"kubernetes.io/projected/c5474343-621e-4dd9-948c-ad71bd6f00c2-kube-api-access-s62xs\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.821277 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820641 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59643abf-923a-4d4d-8235-c2ad22c8aea0-hosts-file\") pod \"node-resolver-kzsn2\" (UID: \"59643abf-923a-4d4d-8235-c2ad22c8aea0\") " pod="openshift-dns/node-resolver-kzsn2" Nov 03 20:28:09.821277 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820657 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfrc5\" (UniqueName: \"kubernetes.io/projected/59643abf-923a-4d4d-8235-c2ad22c8aea0-kube-api-access-wfrc5\") pod \"node-resolver-kzsn2\" (UID: \"59643abf-923a-4d4d-8235-c2ad22c8aea0\") " pod="openshift-dns/node-resolver-kzsn2" Nov 03 20:28:09.821277 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820715 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-slash\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.821277 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820745 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-node-log\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.821277 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820771 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-cni-netd\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.821277 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820801 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmnw9\" (UniqueName: \"kubernetes.io/projected/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-kube-api-access-bmnw9\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.821277 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820838 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-sys\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.821277 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820874 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-sys-fs\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:09.821277 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820905 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-var-lib-openvswitch\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.821277 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820937 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-cnibin\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.821277 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820964 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrs5m\" (UniqueName: \"kubernetes.io/projected/9844c086-efc5-46f3-9d37-34c2fa013f34-kube-api-access-zrs5m\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:09.821277 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.820990 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef30d813-b7ed-417d-a9fe-85118bf77a95-dbus\") pod \"global-pull-secret-syncer-2sjv9\" (UID: \"ef30d813-b7ed-417d-a9fe-85118bf77a95\") " pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:09.821762 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.821086 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-sysctl-conf\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.821762 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.821103 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ss5t\" (UniqueName: \"kubernetes.io/projected/f7ab819a-4321-4958-a5d5-230cbdbd9428-kube-api-access-8ss5t\") pod \"node-ca-5tx7t\" (UID: \"f7ab819a-4321-4958-a5d5-230cbdbd9428\") " pod="openshift-image-registry/node-ca-5tx7t" Nov 03 20:28:09.821762 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.821120 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-log-socket\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.821762 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.821166 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef30d813-b7ed-417d-a9fe-85118bf77a95-dbus\") pod \"global-pull-secret-syncer-2sjv9\" (UID: \"ef30d813-b7ed-417d-a9fe-85118bf77a95\") " pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:09.834503 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.834465 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal" event={"ID":"cb4d50458eb702ff22ca2222c359eed8","Type":"ContainerStarted","Data":"b0742edfa33aa6021b9dc3a08e45c023e436ce141fd7b15572b3831f1a5a7c66"} Nov 03 20:28:09.835537 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.835517 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal" event={"ID":"733c8a62db18973abc6d8fb96bb89db1","Type":"ContainerStarted","Data":"0c5b6a1b1f30d3e9122ef65b5a92319eaecd4674ea7acfb1194edbbb5ea43818"} Nov 03 20:28:09.862676 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.862648 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 20:28:09.862776 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.862704 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b6cd6" Nov 03 20:28:09.870342 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.870319 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 20:28:09.870557 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.870534 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 20:28:09.870665 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.870556 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mlgd9\"" Nov 03 20:28:09.870665 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.870554 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 20:28:09.870794 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.870676 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:28:09.870794 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.870684 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 20:28:09.870905 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.870808 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 20:28:09.870905 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.870850 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 20:28:09.872095 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.871803 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 20:28:09.872095 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.871843 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 20:28:09.872095 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.871890 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 20:28:09.872095 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.871920 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 20:28:09.872095 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.871940 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 20:28:09.872095 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.871947 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 20:28:09.872095 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.872094 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 20:28:09.872493 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.872264 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c6vtd\"" Nov 03 20:28:09.872493 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.872379 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w48hg\"" Nov 03 20:28:09.872493 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.872432 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 20:28:09.872493 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.872464 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 20:28:09.881832 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.881811 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jb4cj\"" Nov 03 20:28:09.881832 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.881833 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-d54s8\"" Nov 03 20:28:09.917905 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.917883 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-w9qmc\"" Nov 03 20:28:09.917995 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.917944 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 20:28:09.921428 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921413 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-sysctl-d\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.921493 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921436 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7ab819a-4321-4958-a5d5-230cbdbd9428-host\") pod \"node-ca-5tx7t\" (UID: \"f7ab819a-4321-4958-a5d5-230cbdbd9428\") " pod="openshift-image-registry/node-ca-5tx7t" Nov 03 20:28:09.921493 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921454 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-socket-dir\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:09.921580 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921501 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-registration-dir\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:09.921580 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921512 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7ab819a-4321-4958-a5d5-230cbdbd9428-host\") pod \"node-ca-5tx7t\" (UID: \"f7ab819a-4321-4958-a5d5-230cbdbd9428\") " pod="openshift-image-registry/node-ca-5tx7t" Nov 03 20:28:09.921580 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921530 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-systemd-units\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.921580 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921560 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-ovn-node-metrics-cert\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.921770 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921590 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-system-cni-dir\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.921770 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921596 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-systemd-units\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.921770 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921629 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-sysctl-d\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.921770 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921622 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4cc66ac6-6311-4144-8db1-3c39182e1a19-iptables-alerter-script\") pod \"iptables-alerter-bjbkk\" (UID: \"4cc66ac6-6311-4144-8db1-3c39182e1a19\") " pod="openshift-network-operator/iptables-alerter-bjbkk" Nov 03 20:28:09.921770 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921680 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-kubelet\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.921770 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921690 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-socket-dir\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:09.921770 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921758 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.922101 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921793 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-var-lib-cni-bin\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.922101 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921772 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-registration-dir\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:09.922101 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921823 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-multus-conf-dir\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.922101 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921849 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrlqh\" (UniqueName: \"kubernetes.io/projected/d5125fef-6a2d-45d6-99d2-90915b575f32-kube-api-access-hrlqh\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.922101 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921853 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-var-lib-cni-bin\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.922101 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921857 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.922101 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921874 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-run\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.922101 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921877 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-multus-conf-dir\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.922101 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921900 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-etc-openvswitch\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.922101 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921923 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-run\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.922101 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921934 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-run-k8s-cni-cncf-io\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.922101 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921940 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-etc-openvswitch\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.922101 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921969 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-var-lib-cni-multus\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.922101 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921976 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-run-k8s-cni-cncf-io\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.922101 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.921994 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e99c666f-7bb3-4856-a2fc-935ee184dae3-konnectivity-ca\") pod \"konnectivity-agent-ccxwr\" (UID: \"e99c666f-7bb3-4856-a2fc-935ee184dae3\") " pod="kube-system/konnectivity-agent-ccxwr" Nov 03 20:28:09.922101 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922018 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-cni-bin\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.922101 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922020 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-var-lib-cni-multus\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.922101 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922036 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-system-cni-dir\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.922871 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922045 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59643abf-923a-4d4d-8235-c2ad22c8aea0-tmp-dir\") pod \"node-resolver-kzsn2\" (UID: \"59643abf-923a-4d4d-8235-c2ad22c8aea0\") " pod="openshift-dns/node-resolver-kzsn2" Nov 03 20:28:09.922871 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922063 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-kubelet\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.922871 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922069 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-run-openvswitch\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.922871 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922101 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-ovnkube-config\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.922871 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922107 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-cni-bin\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.922871 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922144 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-run-openvswitch\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.922871 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922173 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5125fef-6a2d-45d6-99d2-90915b575f32-cni-binary-copy\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.922871 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922190 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-var-lib-kubelet\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.922871 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922207 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-etc-selinux\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:09.922871 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922223 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-run-ovn\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.922871 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922239 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-run-ovn-kubernetes\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.922871 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922263 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdhpf\" (UniqueName: \"kubernetes.io/projected/4cc66ac6-6311-4144-8db1-3c39182e1a19-kube-api-access-vdhpf\") pod \"iptables-alerter-bjbkk\" (UID: \"4cc66ac6-6311-4144-8db1-3c39182e1a19\") " pod="openshift-network-operator/iptables-alerter-bjbkk" Nov 03 20:28:09.922871 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922281 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:09.922871 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922297 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fhqp\" (UniqueName: \"kubernetes.io/projected/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-kube-api-access-4fhqp\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:09.922871 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922355 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-run-ovn\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.922871 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922392 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-run-multus-certs\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.922871 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922423 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-systemd\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.923684 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922427 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-etc-selinux\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:09.923684 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922439 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-var-lib-kubelet\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.923684 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922456 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-tuned\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.923684 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922497 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e99c666f-7bb3-4856-a2fc-935ee184dae3-agent-certs\") pod \"konnectivity-agent-ccxwr\" (UID: \"e99c666f-7bb3-4856-a2fc-935ee184dae3\") " pod="kube-system/konnectivity-agent-ccxwr" Nov 03 20:28:09.923684 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922509 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-run-multus-certs\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.923684 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922516 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-multus-cni-dir\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.923684 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922533 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:09.923684 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922549 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-var-lib-kubelet\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.923684 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922556 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-multus-cni-dir\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.923684 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922568 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-run-ovn-kubernetes\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.923684 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922472 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-systemd\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.923684 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922583 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-hostroot\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.923684 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922590 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-var-lib-kubelet\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.923684 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922608 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-sysconfig\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.923684 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922649 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-lib-modules\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.923684 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922647 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-hostroot\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.923684 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922674 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-multus-socket-dir-parent\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.923684 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922703 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-modprobe-d\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.924462 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922711 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-sysconfig\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.924462 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922729 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-kubernetes\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.924462 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922767 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-multus-socket-dir-parent\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.924462 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922754 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5474343-621e-4dd9-948c-ad71bd6f00c2-tmp\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.924462 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922802 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-lib-modules\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.924462 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922805 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-ovnkube-script-lib\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.924462 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922826 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-modprobe-d\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.924462 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922849 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-etc-kubernetes\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.924462 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922876 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cc66ac6-6311-4144-8db1-3c39182e1a19-host-slash\") pod \"iptables-alerter-bjbkk\" (UID: \"4cc66ac6-6311-4144-8db1-3c39182e1a19\") " pod="openshift-network-operator/iptables-alerter-bjbkk" Nov 03 20:28:09.924462 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922901 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-device-dir\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:09.924462 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922925 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-run-netns\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.924462 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922946 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-env-overrides\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.924462 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922969 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s62xs\" (UniqueName: \"kubernetes.io/projected/c5474343-621e-4dd9-948c-ad71bd6f00c2-kube-api-access-s62xs\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.924462 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.922995 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59643abf-923a-4d4d-8235-c2ad22c8aea0-hosts-file\") pod \"node-resolver-kzsn2\" (UID: \"59643abf-923a-4d4d-8235-c2ad22c8aea0\") " pod="openshift-dns/node-resolver-kzsn2" Nov 03 20:28:09.924462 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923004 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-kubernetes\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.924462 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923024 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfrc5\" (UniqueName: \"kubernetes.io/projected/59643abf-923a-4d4d-8235-c2ad22c8aea0-kube-api-access-wfrc5\") pod \"node-resolver-kzsn2\" (UID: \"59643abf-923a-4d4d-8235-c2ad22c8aea0\") " pod="openshift-dns/node-resolver-kzsn2" Nov 03 20:28:09.924462 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923049 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-slash\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.924462 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923071 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-node-log\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.925584 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923079 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-run-netns\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.925584 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923097 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-cni-netd\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.925584 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923109 2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 20:28:09.925584 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923124 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmnw9\" (UniqueName: \"kubernetes.io/projected/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-kube-api-access-bmnw9\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.925584 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923138 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-etc-kubernetes\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.925584 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923151 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-sys\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.925584 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923184 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59643abf-923a-4d4d-8235-c2ad22c8aea0-tmp-dir\") pod \"node-resolver-kzsn2\" (UID: \"59643abf-923a-4d4d-8235-c2ad22c8aea0\") " pod="openshift-dns/node-resolver-kzsn2" Nov 03 20:28:09.925584 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923188 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-sys-fs\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:09.925584 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923225 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-var-lib-openvswitch\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.925584 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923225 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4cc66ac6-6311-4144-8db1-3c39182e1a19-iptables-alerter-script\") pod \"iptables-alerter-bjbkk\" (UID: \"4cc66ac6-6311-4144-8db1-3c39182e1a19\") " pod="openshift-network-operator/iptables-alerter-bjbkk" Nov 03 20:28:09.925584 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923266 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-device-dir\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:09.925584 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923273 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-node-log\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.925584 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923328 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-sys\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.925584 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923433 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-cni-netd\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.925584 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923439 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e99c666f-7bb3-4856-a2fc-935ee184dae3-konnectivity-ca\") pod \"konnectivity-agent-ccxwr\" (UID: \"e99c666f-7bb3-4856-a2fc-935ee184dae3\") " pod="kube-system/konnectivity-agent-ccxwr" Nov 03 20:28:09.925584 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923518 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-slash\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.925584 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923529 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-var-lib-openvswitch\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.925584 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923530 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5125fef-6a2d-45d6-99d2-90915b575f32-cni-binary-copy\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.926487 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923551 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-cnibin\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.926487 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923580 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59643abf-923a-4d4d-8235-c2ad22c8aea0-hosts-file\") pod \"node-resolver-kzsn2\" (UID: \"59643abf-923a-4d4d-8235-c2ad22c8aea0\") " pod="openshift-dns/node-resolver-kzsn2" Nov 03 20:28:09.926487 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923581 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-sysctl-conf\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.926487 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923629 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-sys-fs\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:09.926487 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923618 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ss5t\" (UniqueName: \"kubernetes.io/projected/f7ab819a-4321-4958-a5d5-230cbdbd9428-kube-api-access-8ss5t\") pod \"node-ca-5tx7t\" (UID: \"f7ab819a-4321-4958-a5d5-230cbdbd9428\") " pod="openshift-image-registry/node-ca-5tx7t" Nov 03 20:28:09.926487 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923689 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-log-socket\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.926487 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923716 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-sysctl-conf\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.926487 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923727 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-host\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.926487 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923764 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f7ab819a-4321-4958-a5d5-230cbdbd9428-serviceca\") pod \"node-ca-5tx7t\" (UID: \"f7ab819a-4321-4958-a5d5-230cbdbd9428\") " pod="openshift-image-registry/node-ca-5tx7t" Nov 03 20:28:09.926487 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923779 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-host\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.926487 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923791 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-run-systemd\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.926487 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923816 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-run-netns\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.926487 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923850 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5125fef-6a2d-45d6-99d2-90915b575f32-multus-daemon-config\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.926487 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923983 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-log-socket\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.926487 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.923191 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cc66ac6-6311-4144-8db1-3c39182e1a19-host-slash\") pod \"iptables-alerter-bjbkk\" (UID: \"4cc66ac6-6311-4144-8db1-3c39182e1a19\") " pod="openshift-network-operator/iptables-alerter-bjbkk" Nov 03 20:28:09.926487 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.924024 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-run-systemd\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.926487 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.924053 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-cnibin\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.926487 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.924063 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-run-netns\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.927334 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.924573 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-env-overrides\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.927334 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.924722 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-ovnkube-script-lib\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.927334 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.924762 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-ovnkube-config\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.927334 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:09.925636 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:28:09.927334 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.925653 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5125fef-6a2d-45d6-99d2-90915b575f32-multus-daemon-config\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.927334 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:09.925662 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:28:09.927334 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:09.925722 2572 projected.go:196] Error preparing data for projected volume kube-api-access-6nlpn for pod openshift-network-diagnostics/network-check-target-l4pbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:09.927334 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:09.925800 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6cd2ab07-4196-4ed1-b357-38f01b99b71d-kube-api-access-6nlpn podName:6cd2ab07-4196-4ed1-b357-38f01b99b71d nodeName:}" failed. No retries permitted until 2026-11-03 20:28:10.425782171 +0000 UTC m=+3.222881724 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6nlpn" (UniqueName: "kubernetes.io/projected/6cd2ab07-4196-4ed1-b357-38f01b99b71d-kube-api-access-6nlpn") pod "network-check-target-l4pbv" (UID: "6cd2ab07-4196-4ed1-b357-38f01b99b71d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:09.927334 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.926135 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f7ab819a-4321-4958-a5d5-230cbdbd9428-serviceca\") pod \"node-ca-5tx7t\" (UID: \"f7ab819a-4321-4958-a5d5-230cbdbd9428\") " pod="openshift-image-registry/node-ca-5tx7t" Nov 03 20:28:09.927659 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.927521 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5474343-621e-4dd9-948c-ad71bd6f00c2-tmp\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.927659 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.927595 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-tuned\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.927826 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.927803 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-ovn-node-metrics-cert\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.928386 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.928363 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e99c666f-7bb3-4856-a2fc-935ee184dae3-agent-certs\") pod \"konnectivity-agent-ccxwr\" (UID: \"e99c666f-7bb3-4856-a2fc-935ee184dae3\") " pod="kube-system/konnectivity-agent-ccxwr" Nov 03 20:28:09.928594 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.928577 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9487d\" (UniqueName: \"kubernetes.io/projected/2281c3da-747c-4a37-89ec-b764d8a95405-kube-api-access-9487d\") pod \"network-metrics-daemon-f28pp\" (UID: \"2281c3da-747c-4a37-89ec-b764d8a95405\") " pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:09.928911 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.928894 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrs5m\" (UniqueName: \"kubernetes.io/projected/9844c086-efc5-46f3-9d37-34c2fa013f34-kube-api-access-zrs5m\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:09.986823 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.986755 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fhqp\" (UniqueName: \"kubernetes.io/projected/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-kube-api-access-4fhqp\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:09.987110 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.987088 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfrc5\" (UniqueName: \"kubernetes.io/projected/59643abf-923a-4d4d-8235-c2ad22c8aea0-kube-api-access-wfrc5\") pod \"node-resolver-kzsn2\" (UID: \"59643abf-923a-4d4d-8235-c2ad22c8aea0\") " pod="openshift-dns/node-resolver-kzsn2" Nov 03 20:28:09.988330 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.988306 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdhpf\" (UniqueName: \"kubernetes.io/projected/4cc66ac6-6311-4144-8db1-3c39182e1a19-kube-api-access-vdhpf\") pod \"iptables-alerter-bjbkk\" (UID: \"4cc66ac6-6311-4144-8db1-3c39182e1a19\") " pod="openshift-network-operator/iptables-alerter-bjbkk" Nov 03 20:28:09.988330 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.988318 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s62xs\" (UniqueName: \"kubernetes.io/projected/c5474343-621e-4dd9-948c-ad71bd6f00c2-kube-api-access-s62xs\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:09.988478 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.988308 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmnw9\" (UniqueName: \"kubernetes.io/projected/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-kube-api-access-bmnw9\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:09.988478 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.988310 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrlqh\" (UniqueName: \"kubernetes.io/projected/d5125fef-6a2d-45d6-99d2-90915b575f32-kube-api-access-hrlqh\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:09.988478 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:09.988310 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ss5t\" (UniqueName: \"kubernetes.io/projected/f7ab819a-4321-4958-a5d5-230cbdbd9428-kube-api-access-8ss5t\") pod \"node-ca-5tx7t\" (UID: \"f7ab819a-4321-4958-a5d5-230cbdbd9428\") " pod="openshift-image-registry/node-ca-5tx7t" Nov 03 20:28:10.029071 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:10.029039 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:10.035896 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:10.035868 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:10.039979 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:10.039959 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:10.040074 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:10.040047 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kzsn2" Nov 03 20:28:10.040389 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:10.040361 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9844c086_efc5_46f3_9d37_34c2fa013f34.slice/crio-e0be4d66f0bdf0aac69f183f27d9edb3ba94d26257901efec80d7d7c8eba467e WatchSource:0}: Error finding container e0be4d66f0bdf0aac69f183f27d9edb3ba94d26257901efec80d7d7c8eba467e: Status 404 returned error can't find the container with id e0be4d66f0bdf0aac69f183f27d9edb3ba94d26257901efec80d7d7c8eba467e Nov 03 20:28:10.045432 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:10.045335 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5474343_621e_4dd9_948c_ad71bd6f00c2.slice/crio-18017f4abbe5744c656b764e2974eb9099e93adcb91ae095b384a99ca6a32415 WatchSource:0}: Error finding container 18017f4abbe5744c656b764e2974eb9099e93adcb91ae095b384a99ca6a32415: Status 404 returned error can't find the container with id 18017f4abbe5744c656b764e2974eb9099e93adcb91ae095b384a99ca6a32415 Nov 03 20:28:10.051565 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:10.051544 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3868c3d_8d3a_4c40_8821_ca116f8dcfed.slice/crio-92af30685c6d43145a57276b4e57ebc43fb507c107b145fddbf83cd0120d7867 WatchSource:0}: Error finding container 92af30685c6d43145a57276b4e57ebc43fb507c107b145fddbf83cd0120d7867: Status 404 returned error can't find the container with id 92af30685c6d43145a57276b4e57ebc43fb507c107b145fddbf83cd0120d7867 Nov 03 20:28:10.052514 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:10.052494 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ccxwr" Nov 03 20:28:10.053639 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:10.053610 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59643abf_923a_4d4d_8235_c2ad22c8aea0.slice/crio-a852d31c4b868d05ab7e57a0691f02e1a5c4db0da405f0896605437528d9955c WatchSource:0}: Error finding container a852d31c4b868d05ab7e57a0691f02e1a5c4db0da405f0896605437528d9955c: Status 404 returned error can't find the container with id a852d31c4b868d05ab7e57a0691f02e1a5c4db0da405f0896605437528d9955c Nov 03 20:28:10.054840 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:10.054817 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bjbkk" Nov 03 20:28:10.062015 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:10.061990 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:10.063590 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:10.063564 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99c666f_7bb3_4856_a2fc_935ee184dae3.slice/crio-b7b7f04d8d2fa36cc327714791ba9c625aa82b9acd636b3978179d64ddc92f70 WatchSource:0}: Error finding container b7b7f04d8d2fa36cc327714791ba9c625aa82b9acd636b3978179d64ddc92f70: Status 404 returned error can't find the container with id b7b7f04d8d2fa36cc327714791ba9c625aa82b9acd636b3978179d64ddc92f70 Nov 03 20:28:10.064484 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:10.064465 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5tx7t" Nov 03 20:28:10.065964 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:10.065826 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc66ac6_6311_4144_8db1_3c39182e1a19.slice/crio-764a370bdff5d09b8d06ef47240c6223aae800dac58d02e400dd722008e4da8e WatchSource:0}: Error finding container 764a370bdff5d09b8d06ef47240c6223aae800dac58d02e400dd722008e4da8e: Status 404 returned error can't find the container with id 764a370bdff5d09b8d06ef47240c6223aae800dac58d02e400dd722008e4da8e Nov 03 20:28:10.066665 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:10.066649 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pv8vt" Nov 03 20:28:10.072639 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:10.072613 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48c0b65_db57_43b2_8e7c_6e9e8feddf89.slice/crio-195d211fa9105149de97f3e991e074de3fbefa75aa528ef0e4604e3b4848fca3 WatchSource:0}: Error finding container 195d211fa9105149de97f3e991e074de3fbefa75aa528ef0e4604e3b4848fca3: Status 404 returned error can't find the container with id 195d211fa9105149de97f3e991e074de3fbefa75aa528ef0e4604e3b4848fca3 Nov 03 20:28:10.076596 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:10.076562 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7ab819a_4321_4958_a5d5_230cbdbd9428.slice/crio-b7ec26740a2688262054a179f78c6e66b34689da2e54e2cabb3df4d1b835f026 WatchSource:0}: Error finding container b7ec26740a2688262054a179f78c6e66b34689da2e54e2cabb3df4d1b835f026: Status 404 returned error can't find the container with id b7ec26740a2688262054a179f78c6e66b34689da2e54e2cabb3df4d1b835f026 Nov 03 20:28:10.077231 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:10.077209 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5125fef_6a2d_45d6_99d2_90915b575f32.slice/crio-752a7a52453719ea41ed2a9926482b8016ac42c1c089927f0d5a4a92fec9d9f2 WatchSource:0}: Error finding container 752a7a52453719ea41ed2a9926482b8016ac42c1c089927f0d5a4a92fec9d9f2: Status 404 returned error can't find the container with id 752a7a52453719ea41ed2a9926482b8016ac42c1c089927f0d5a4a92fec9d9f2 Nov 03 20:28:10.327246 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:10.327213 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs\") pod \"network-metrics-daemon-f28pp\" (UID: \"2281c3da-747c-4a37-89ec-b764d8a95405\") " pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:10.327451 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:10.327276 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef30d813-b7ed-417d-a9fe-85118bf77a95-original-pull-secret\") pod \"global-pull-secret-syncer-2sjv9\" (UID: \"ef30d813-b7ed-417d-a9fe-85118bf77a95\") " pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:10.327451 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:10.327334 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:10.327451 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:10.327405 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:10.327451 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:10.327423 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs podName:2281c3da-747c-4a37-89ec-b764d8a95405 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:11.327404166 +0000 UTC m=+4.124503718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs") pod "network-metrics-daemon-f28pp" (UID: "2281c3da-747c-4a37-89ec-b764d8a95405") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:10.327451 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:10.327441 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef30d813-b7ed-417d-a9fe-85118bf77a95-original-pull-secret podName:ef30d813-b7ed-417d-a9fe-85118bf77a95 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:11.327429891 +0000 UTC m=+4.124529422 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef30d813-b7ed-417d-a9fe-85118bf77a95-original-pull-secret") pod "global-pull-secret-syncer-2sjv9" (UID: "ef30d813-b7ed-417d-a9fe-85118bf77a95") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:10.428020 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:10.427987 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nlpn\" (UniqueName: \"kubernetes.io/projected/6cd2ab07-4196-4ed1-b357-38f01b99b71d-kube-api-access-6nlpn\") pod \"network-check-target-l4pbv\" (UID: \"6cd2ab07-4196-4ed1-b357-38f01b99b71d\") " pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:10.428225 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:10.428161 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:28:10.428225 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:10.428179 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:28:10.428225 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:10.428189 2572 projected.go:196] Error preparing data for projected volume kube-api-access-6nlpn for pod openshift-network-diagnostics/network-check-target-l4pbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:10.428340 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:10.428242 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6cd2ab07-4196-4ed1-b357-38f01b99b71d-kube-api-access-6nlpn podName:6cd2ab07-4196-4ed1-b357-38f01b99b71d nodeName:}" failed. No retries permitted until 2026-11-03 20:28:11.428226765 +0000 UTC m=+4.225326309 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6nlpn" (UniqueName: "kubernetes.io/projected/6cd2ab07-4196-4ed1-b357-38f01b99b71d-kube-api-access-6nlpn") pod "network-check-target-l4pbv" (UID: "6cd2ab07-4196-4ed1-b357-38f01b99b71d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:10.838899 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:10.838857 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tl7xc" event={"ID":"9844c086-efc5-46f3-9d37-34c2fa013f34","Type":"ContainerStarted","Data":"e0be4d66f0bdf0aac69f183f27d9edb3ba94d26257901efec80d7d7c8eba467e"} Nov 03 20:28:10.840739 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:10.840685 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5tx7t" event={"ID":"f7ab819a-4321-4958-a5d5-230cbdbd9428","Type":"ContainerStarted","Data":"b7ec26740a2688262054a179f78c6e66b34689da2e54e2cabb3df4d1b835f026"} Nov 03 20:28:10.842686 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:10.842540 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82vml" event={"ID":"d48c0b65-db57-43b2-8e7c-6e9e8feddf89","Type":"ContainerStarted","Data":"195d211fa9105149de97f3e991e074de3fbefa75aa528ef0e4604e3b4848fca3"} Nov 03 20:28:10.844857 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:10.844809 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bjbkk" event={"ID":"4cc66ac6-6311-4144-8db1-3c39182e1a19","Type":"ContainerStarted","Data":"764a370bdff5d09b8d06ef47240c6223aae800dac58d02e400dd722008e4da8e"} Nov 03 20:28:10.846476 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:10.846413 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ccxwr" event={"ID":"e99c666f-7bb3-4856-a2fc-935ee184dae3","Type":"ContainerStarted","Data":"b7b7f04d8d2fa36cc327714791ba9c625aa82b9acd636b3978179d64ddc92f70"} Nov 03 20:28:10.848073 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:10.848045 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kzsn2" event={"ID":"59643abf-923a-4d4d-8235-c2ad22c8aea0","Type":"ContainerStarted","Data":"a852d31c4b868d05ab7e57a0691f02e1a5c4db0da405f0896605437528d9955c"} Nov 03 20:28:10.849429 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:10.849404 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pv8vt" event={"ID":"d5125fef-6a2d-45d6-99d2-90915b575f32","Type":"ContainerStarted","Data":"752a7a52453719ea41ed2a9926482b8016ac42c1c089927f0d5a4a92fec9d9f2"} Nov 03 20:28:10.851396 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:10.851373 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" event={"ID":"a3868c3d-8d3a-4c40-8821-ca116f8dcfed","Type":"ContainerStarted","Data":"92af30685c6d43145a57276b4e57ebc43fb507c107b145fddbf83cd0120d7867"} Nov 03 20:28:10.852893 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:10.852868 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" event={"ID":"c5474343-621e-4dd9-948c-ad71bd6f00c2","Type":"ContainerStarted","Data":"18017f4abbe5744c656b764e2974eb9099e93adcb91ae095b384a99ca6a32415"} Nov 03 20:28:10.865840 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:10.865798 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:23:10 +0000 UTC" deadline="2028-04-01 10:26:47.376306318 +0000 UTC" Nov 03 20:28:10.865840 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:10.865839 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12349h58m36.510470435s" Nov 03 20:28:11.336307 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:11.336267 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs\") pod \"network-metrics-daemon-f28pp\" (UID: \"2281c3da-747c-4a37-89ec-b764d8a95405\") " pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:11.336509 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:11.336344 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef30d813-b7ed-417d-a9fe-85118bf77a95-original-pull-secret\") pod \"global-pull-secret-syncer-2sjv9\" (UID: \"ef30d813-b7ed-417d-a9fe-85118bf77a95\") " pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:11.336509 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:11.336505 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:11.336641 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:11.336576 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef30d813-b7ed-417d-a9fe-85118bf77a95-original-pull-secret podName:ef30d813-b7ed-417d-a9fe-85118bf77a95 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:13.336556014 +0000 UTC m=+6.133655549 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef30d813-b7ed-417d-a9fe-85118bf77a95-original-pull-secret") pod "global-pull-secret-syncer-2sjv9" (UID: "ef30d813-b7ed-417d-a9fe-85118bf77a95") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:11.336752 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:11.336655 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:11.336752 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:11.336695 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs podName:2281c3da-747c-4a37-89ec-b764d8a95405 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:13.336681433 +0000 UTC m=+6.133780969 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs") pod "network-metrics-daemon-f28pp" (UID: "2281c3da-747c-4a37-89ec-b764d8a95405") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:11.437182 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:11.437142 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nlpn\" (UniqueName: \"kubernetes.io/projected/6cd2ab07-4196-4ed1-b357-38f01b99b71d-kube-api-access-6nlpn\") pod \"network-check-target-l4pbv\" (UID: \"6cd2ab07-4196-4ed1-b357-38f01b99b71d\") " pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:11.437426 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:11.437386 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:28:11.437426 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:11.437414 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:28:11.437426 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:11.437428 2572 projected.go:196] Error preparing data for projected volume kube-api-access-6nlpn for pod openshift-network-diagnostics/network-check-target-l4pbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:11.437647 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:11.437485 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6cd2ab07-4196-4ed1-b357-38f01b99b71d-kube-api-access-6nlpn podName:6cd2ab07-4196-4ed1-b357-38f01b99b71d nodeName:}" failed. No retries permitted until 2026-11-03 20:28:13.437466962 +0000 UTC m=+6.234566510 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6nlpn" (UniqueName: "kubernetes.io/projected/6cd2ab07-4196-4ed1-b357-38f01b99b71d-kube-api-access-6nlpn") pod "network-check-target-l4pbv" (UID: "6cd2ab07-4196-4ed1-b357-38f01b99b71d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:11.831841 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:11.831562 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:11.831841 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:11.831686 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f28pp" podUID="2281c3da-747c-4a37-89ec-b764d8a95405" Nov 03 20:28:11.831841 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:11.831754 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:11.831841 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:11.831802 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sjv9" podUID="ef30d813-b7ed-417d-a9fe-85118bf77a95" Nov 03 20:28:11.832280 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:11.832180 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:11.832280 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:11.832252 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4pbv" podUID="6cd2ab07-4196-4ed1-b357-38f01b99b71d" Nov 03 20:28:11.866566 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:11.866505 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:23:10 +0000 UTC" deadline="2028-05-24 05:43:31.401808547 +0000 UTC" Nov 03 20:28:11.866566 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:11.866544 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13617h15m19.535269392s" Nov 03 20:28:13.354703 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:13.354206 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs\") pod \"network-metrics-daemon-f28pp\" (UID: \"2281c3da-747c-4a37-89ec-b764d8a95405\") " pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:13.354703 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:13.354281 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef30d813-b7ed-417d-a9fe-85118bf77a95-original-pull-secret\") pod \"global-pull-secret-syncer-2sjv9\" (UID: \"ef30d813-b7ed-417d-a9fe-85118bf77a95\") " pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:13.354703 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:13.354477 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:13.354703 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:13.354549 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef30d813-b7ed-417d-a9fe-85118bf77a95-original-pull-secret podName:ef30d813-b7ed-417d-a9fe-85118bf77a95 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:17.354526208 +0000 UTC m=+10.151625752 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef30d813-b7ed-417d-a9fe-85118bf77a95-original-pull-secret") pod "global-pull-secret-syncer-2sjv9" (UID: "ef30d813-b7ed-417d-a9fe-85118bf77a95") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:13.354703 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:13.354620 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:13.354703 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:13.354657 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs podName:2281c3da-747c-4a37-89ec-b764d8a95405 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:17.354644542 +0000 UTC m=+10.151744088 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs") pod "network-metrics-daemon-f28pp" (UID: "2281c3da-747c-4a37-89ec-b764d8a95405") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:13.455568 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:13.455526 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nlpn\" (UniqueName: \"kubernetes.io/projected/6cd2ab07-4196-4ed1-b357-38f01b99b71d-kube-api-access-6nlpn\") pod \"network-check-target-l4pbv\" (UID: \"6cd2ab07-4196-4ed1-b357-38f01b99b71d\") " pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:13.455786 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:13.455765 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:28:13.455875 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:13.455797 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:28:13.455875 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:13.455811 2572 projected.go:196] Error preparing data for projected volume kube-api-access-6nlpn for pod openshift-network-diagnostics/network-check-target-l4pbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:13.455875 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:13.455872 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6cd2ab07-4196-4ed1-b357-38f01b99b71d-kube-api-access-6nlpn podName:6cd2ab07-4196-4ed1-b357-38f01b99b71d nodeName:}" failed. No retries permitted until 2026-11-03 20:28:17.455852509 +0000 UTC m=+10.252952057 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6nlpn" (UniqueName: "kubernetes.io/projected/6cd2ab07-4196-4ed1-b357-38f01b99b71d-kube-api-access-6nlpn") pod "network-check-target-l4pbv" (UID: "6cd2ab07-4196-4ed1-b357-38f01b99b71d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:13.833750 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:13.833719 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:13.833945 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:13.833862 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f28pp" podUID="2281c3da-747c-4a37-89ec-b764d8a95405" Nov 03 20:28:13.833945 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:13.833927 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:13.834139 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:13.833999 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sjv9" podUID="ef30d813-b7ed-417d-a9fe-85118bf77a95" Nov 03 20:28:13.834139 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:13.834045 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:13.834139 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:13.834115 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4pbv" podUID="6cd2ab07-4196-4ed1-b357-38f01b99b71d" Nov 03 20:28:14.880280 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:14.879345 2572 generic.go:356] "Generic (PLEG): container finished" podID="cb4d50458eb702ff22ca2222c359eed8" containerID="5615c1bf4299b681a28cedf9ab1bec91cd637adea9348e9fb47e1a89edc518f1" exitCode=0 Nov 03 20:28:14.880280 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:14.879463 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal" event={"ID":"cb4d50458eb702ff22ca2222c359eed8","Type":"ContainerDied","Data":"5615c1bf4299b681a28cedf9ab1bec91cd637adea9348e9fb47e1a89edc518f1"} Nov 03 20:28:14.884958 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:14.884926 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal" event={"ID":"733c8a62db18973abc6d8fb96bb89db1","Type":"ContainerStarted","Data":"0a105902a4f8ede40be45e8206e2bf89c53193a0c115daa302d8aacb7729dacb"} Nov 03 20:28:14.894526 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:14.894491 2572 generic.go:356] "Generic (PLEG): container finished" podID="9844c086-efc5-46f3-9d37-34c2fa013f34" containerID="fa97359fa56a27ebfe28bbb6de377e7daa3bc7bb815d3369a38c01424ee04d6a" exitCode=0 Nov 03 20:28:14.894616 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:14.894533 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tl7xc" event={"ID":"9844c086-efc5-46f3-9d37-34c2fa013f34","Type":"ContainerDied","Data":"fa97359fa56a27ebfe28bbb6de377e7daa3bc7bb815d3369a38c01424ee04d6a"} Nov 03 20:28:15.260831 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:15.260756 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal" podStartSLOduration=6.260736238 podStartE2EDuration="6.260736238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:15.129400284 +0000 UTC m=+7.926499838" watchObservedRunningTime="2026-11-03 20:28:15.260736238 +0000 UTC m=+8.057835790" Nov 03 20:28:15.833116 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:15.833077 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:15.833320 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:15.833224 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f28pp" podUID="2281c3da-747c-4a37-89ec-b764d8a95405" Nov 03 20:28:15.833320 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:15.833295 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:15.833477 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:15.833369 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sjv9" podUID="ef30d813-b7ed-417d-a9fe-85118bf77a95" Nov 03 20:28:15.833477 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:15.833448 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:15.833595 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:15.833517 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4pbv" podUID="6cd2ab07-4196-4ed1-b357-38f01b99b71d" Nov 03 20:28:15.901015 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:15.900292 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal" event={"ID":"cb4d50458eb702ff22ca2222c359eed8","Type":"ContainerStarted","Data":"6098514fcedcaa160b8cd7b6a3872d6cca8720c3bd35197a99a6f08b4b90619c"} Nov 03 20:28:15.905008 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:15.904928 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ccxwr" event={"ID":"e99c666f-7bb3-4856-a2fc-935ee184dae3","Type":"ContainerStarted","Data":"7d18ff7786daf67bc461039fbefe7566a2151eba434c44b5c2ee1a7209ffd5b6"} Nov 03 20:28:16.151758 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:16.151692 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal" podStartSLOduration=7.151672786 podStartE2EDuration="7.151672786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:16.026316768 +0000 UTC m=+8.823416321" watchObservedRunningTime="2026-11-03 20:28:16.151672786 +0000 UTC m=+8.948772340" Nov 03 20:28:16.152046 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:16.152010 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ccxwr" podStartSLOduration=2.958188282 podStartE2EDuration="8.152001641s" totalImagesPullingTime="5.193813359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:16.151642693 +0000 UTC m=+8.948742246" watchObservedRunningTime="2026-11-03 20:28:16.152001641 +0000 UTC m=+8.949101195" Nov 03 20:28:16.911141 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:16.911082 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ccxwr" Nov 03 20:28:16.912315 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:16.912290 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ccxwr" Nov 03 20:28:16.913739 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:16.913712 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5tx7t" event={"ID":"f7ab819a-4321-4958-a5d5-230cbdbd9428","Type":"ContainerStarted","Data":"0844d6032ae4e0b9612cad322400529d6b93338236dd8b1edc0ae56ac3993d34"} Nov 03 20:28:17.391869 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:17.391779 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs\") pod \"network-metrics-daemon-f28pp\" (UID: \"2281c3da-747c-4a37-89ec-b764d8a95405\") " pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:17.391869 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:17.391836 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef30d813-b7ed-417d-a9fe-85118bf77a95-original-pull-secret\") pod \"global-pull-secret-syncer-2sjv9\" (UID: \"ef30d813-b7ed-417d-a9fe-85118bf77a95\") " pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:17.392020 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:17.391967 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:17.392073 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:17.392027 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef30d813-b7ed-417d-a9fe-85118bf77a95-original-pull-secret podName:ef30d813-b7ed-417d-a9fe-85118bf77a95 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:25.392009973 +0000 UTC m=+18.189109508 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef30d813-b7ed-417d-a9fe-85118bf77a95-original-pull-secret") pod "global-pull-secret-syncer-2sjv9" (UID: "ef30d813-b7ed-417d-a9fe-85118bf77a95") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:17.392142 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:17.392092 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:17.392142 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:17.392124 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs podName:2281c3da-747c-4a37-89ec-b764d8a95405 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:25.392113686 +0000 UTC m=+18.189213217 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs") pod "network-metrics-daemon-f28pp" (UID: "2281c3da-747c-4a37-89ec-b764d8a95405") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:17.492973 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:17.492898 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nlpn\" (UniqueName: \"kubernetes.io/projected/6cd2ab07-4196-4ed1-b357-38f01b99b71d-kube-api-access-6nlpn\") pod \"network-check-target-l4pbv\" (UID: \"6cd2ab07-4196-4ed1-b357-38f01b99b71d\") " pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:17.493130 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:17.493075 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:28:17.493130 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:17.493100 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:28:17.493130 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:17.493113 2572 projected.go:196] Error preparing data for projected volume kube-api-access-6nlpn for pod openshift-network-diagnostics/network-check-target-l4pbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:17.493303 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:17.493171 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6cd2ab07-4196-4ed1-b357-38f01b99b71d-kube-api-access-6nlpn podName:6cd2ab07-4196-4ed1-b357-38f01b99b71d nodeName:}" failed. No retries permitted until 2026-11-03 20:28:25.493152926 +0000 UTC m=+18.290252460 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6nlpn" (UniqueName: "kubernetes.io/projected/6cd2ab07-4196-4ed1-b357-38f01b99b71d-kube-api-access-6nlpn") pod "network-check-target-l4pbv" (UID: "6cd2ab07-4196-4ed1-b357-38f01b99b71d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:17.832630 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:17.832597 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:17.832816 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:17.832724 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sjv9" podUID="ef30d813-b7ed-417d-a9fe-85118bf77a95" Nov 03 20:28:17.835060 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:17.835033 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:17.835188 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:17.835136 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f28pp" podUID="2281c3da-747c-4a37-89ec-b764d8a95405" Nov 03 20:28:17.835247 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:17.835188 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:17.835298 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:17.835243 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4pbv" podUID="6cd2ab07-4196-4ed1-b357-38f01b99b71d" Nov 03 20:28:17.919830 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:17.919796 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" event={"ID":"a3868c3d-8d3a-4c40-8821-ca116f8dcfed","Type":"ContainerStarted","Data":"922fdb9051069aa558a831516549c3dd5eedc927380f5764ac173e481bd05c9f"} Nov 03 20:28:17.920227 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:17.919901 2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:28:19.632799 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:19.632773 2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 20:28:19.691446 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:19.691217 2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:28:19.632797003Z","UUID":"ae7b8af7-100e-49d3-840c-6c8fcc3e8602","Handler":null,"Name":"","Endpoint":""} Nov 03 20:28:19.694654 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:19.694634 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 20:28:19.694782 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:19.694667 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 20:28:19.831385 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:19.831291 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:19.831385 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:19.831373 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:19.831546 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:19.831390 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:19.831546 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:19.831510 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sjv9" podUID="ef30d813-b7ed-417d-a9fe-85118bf77a95" Nov 03 20:28:19.831626 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:19.831573 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4pbv" podUID="6cd2ab07-4196-4ed1-b357-38f01b99b71d" Nov 03 20:28:19.831662 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:19.831626 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f28pp" podUID="2281c3da-747c-4a37-89ec-b764d8a95405" Nov 03 20:28:19.926297 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:19.926025 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pv8vt" event={"ID":"d5125fef-6a2d-45d6-99d2-90915b575f32","Type":"ContainerStarted","Data":"bcb80faa822a578ad9b67c7bb31f004e88b5063a6110a2a09d1302e09cde5d51"} Nov 03 20:28:19.929057 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:19.929025 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" event={"ID":"a3868c3d-8d3a-4c40-8821-ca116f8dcfed","Type":"ContainerStarted","Data":"031ea264d06227358e47b9bd9da27a18f31ca9992e3eab82b291418b514fe0bc"} Nov 03 20:28:19.930562 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:19.930540 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kzsn2" event={"ID":"59643abf-923a-4d4d-8235-c2ad22c8aea0","Type":"ContainerStarted","Data":"808480fd2a6a7a8c584476c00ea81862a5f026c4d7065098008bae3e93bd10b0"} Nov 03 20:28:20.041855 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:20.041767 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5tx7t" podStartSLOduration=5.574645329 podStartE2EDuration="12.041749376s" totalImagesPullingTime="6.467104047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:17.163682055 +0000 UTC m=+9.960781609" watchObservedRunningTime="2026-11-03 20:28:20.041749376 +0000 UTC m=+12.838848929" Nov 03 20:28:20.042042 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:20.041903 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-pv8vt" podStartSLOduration=2.842817252 podStartE2EDuration="12.041891381s" totalImagesPullingTime="9.199074129s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:20.041452399 +0000 UTC m=+12.838551948" watchObservedRunningTime="2026-11-03 20:28:20.041891381 +0000 UTC m=+12.838990933" Nov 03 20:28:20.154933 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:20.154883 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-kzsn2" podStartSLOduration=2.926764015 podStartE2EDuration="12.154867765s" totalImagesPullingTime="9.22810375s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:20.154566404 +0000 UTC m=+12.951665956" watchObservedRunningTime="2026-11-03 20:28:20.154867765 +0000 UTC m=+12.951967314" Nov 03 20:28:20.937310 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:20.936005 2572 generic.go:356] "Generic (PLEG): container finished" podID="9844c086-efc5-46f3-9d37-34c2fa013f34" containerID="c82a8cac5d718b43a49f3d4fea60746ec0665422149c90cf621ede8d693a1550" exitCode=0 Nov 03 20:28:20.937310 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:20.936810 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tl7xc" event={"ID":"9844c086-efc5-46f3-9d37-34c2fa013f34","Type":"ContainerDied","Data":"c82a8cac5d718b43a49f3d4fea60746ec0665422149c90cf621ede8d693a1550"} Nov 03 20:28:21.831260 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:21.831072 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:21.831473 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:21.831082 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:21.831473 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:21.831372 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f28pp" podUID="2281c3da-747c-4a37-89ec-b764d8a95405" Nov 03 20:28:21.831473 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:21.831130 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:21.831473 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:21.831448 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4pbv" podUID="6cd2ab07-4196-4ed1-b357-38f01b99b71d" Nov 03 20:28:21.831633 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:21.831524 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sjv9" podUID="ef30d813-b7ed-417d-a9fe-85118bf77a95" Nov 03 20:28:21.940406 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:21.940029 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bjbkk" event={"ID":"4cc66ac6-6311-4144-8db1-3c39182e1a19","Type":"ContainerStarted","Data":"90bffc9b168d29776d0147b97411070f3cfe9b08f99e771024db1b0cd9c2a5bc"} Nov 03 20:28:21.942706 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:21.942678 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" event={"ID":"a3868c3d-8d3a-4c40-8821-ca116f8dcfed","Type":"ContainerStarted","Data":"7972c75c4fabb9291de2b5d87a190529105699696786a9842710be8754545132"} Nov 03 20:28:21.944270 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:21.944244 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" event={"ID":"c5474343-621e-4dd9-948c-ad71bd6f00c2","Type":"ContainerStarted","Data":"fcebff57509cfa4ae2cf08abbd5ef3b069e09316dccf23f1b83de248255bddd0"} Nov 03 20:28:22.054110 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:22.054062 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bjbkk" podStartSLOduration=4.569403546 podStartE2EDuration="14.054050808s" totalImagesPullingTime="9.484647262s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:22.053753094 +0000 UTC m=+14.850852658" watchObservedRunningTime="2026-11-03 20:28:22.054050808 +0000 UTC m=+14.851150357" Nov 03 20:28:22.168932 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:22.168848 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" podStartSLOduration=2.843481681 podStartE2EDuration="14.168833878s" totalImagesPullingTime="11.325352197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:22.168596084 +0000 UTC m=+14.965695634" watchObservedRunningTime="2026-11-03 20:28:22.168833878 +0000 UTC m=+14.965933427" Nov 03 20:28:22.284907 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:22.284136 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" podStartSLOduration=3.526861801 podStartE2EDuration="14.284118135s" totalImagesPullingTime="10.757256334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:22.283800353 +0000 UTC m=+15.080899905" watchObservedRunningTime="2026-11-03 20:28:22.284118135 +0000 UTC m=+15.081217686" Nov 03 20:28:22.949094 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:22.949060 2572 generic.go:356] "Generic (PLEG): container finished" podID="9844c086-efc5-46f3-9d37-34c2fa013f34" containerID="e5b5ef8d438b876180c8e668c3f31f93e4270e40fb5ea9e20ae5190351cc583a" exitCode=0 Nov 03 20:28:22.949771 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:22.949133 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tl7xc" event={"ID":"9844c086-efc5-46f3-9d37-34c2fa013f34","Type":"ContainerDied","Data":"e5b5ef8d438b876180c8e668c3f31f93e4270e40fb5ea9e20ae5190351cc583a"} Nov 03 20:28:23.831613 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:23.831579 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:23.831613 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:23.831604 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:23.831846 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:23.831723 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f28pp" podUID="2281c3da-747c-4a37-89ec-b764d8a95405" Nov 03 20:28:23.831846 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:23.831746 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:23.831846 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:23.831833 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sjv9" podUID="ef30d813-b7ed-417d-a9fe-85118bf77a95" Nov 03 20:28:23.831953 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:23.831892 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4pbv" podUID="6cd2ab07-4196-4ed1-b357-38f01b99b71d" Nov 03 20:28:24.955657 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:24.955624 2572 generic.go:356] "Generic (PLEG): container finished" podID="9844c086-efc5-46f3-9d37-34c2fa013f34" containerID="c446a9d0a01f9762533143bfcdfc18e2bda5724ac90326436d1033ba7a4c7cbc" exitCode=0 Nov 03 20:28:24.956228 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:24.955661 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tl7xc" event={"ID":"9844c086-efc5-46f3-9d37-34c2fa013f34","Type":"ContainerDied","Data":"c446a9d0a01f9762533143bfcdfc18e2bda5724ac90326436d1033ba7a4c7cbc"} Nov 03 20:28:25.458603 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:25.458568 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs\") pod \"network-metrics-daemon-f28pp\" (UID: \"2281c3da-747c-4a37-89ec-b764d8a95405\") " pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:25.458770 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:25.458621 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef30d813-b7ed-417d-a9fe-85118bf77a95-original-pull-secret\") pod \"global-pull-secret-syncer-2sjv9\" (UID: \"ef30d813-b7ed-417d-a9fe-85118bf77a95\") " pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:25.458770 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:25.458681 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:25.458770 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:25.458707 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:25.458770 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:25.458749 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs podName:2281c3da-747c-4a37-89ec-b764d8a95405 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:41.458731153 +0000 UTC m=+34.255830681 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs") pod "network-metrics-daemon-f28pp" (UID: "2281c3da-747c-4a37-89ec-b764d8a95405") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:25.458770 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:25.458763 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef30d813-b7ed-417d-a9fe-85118bf77a95-original-pull-secret podName:ef30d813-b7ed-417d-a9fe-85118bf77a95 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:41.458756705 +0000 UTC m=+34.255856233 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef30d813-b7ed-417d-a9fe-85118bf77a95-original-pull-secret") pod "global-pull-secret-syncer-2sjv9" (UID: "ef30d813-b7ed-417d-a9fe-85118bf77a95") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:25.559506 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:25.559479 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nlpn\" (UniqueName: \"kubernetes.io/projected/6cd2ab07-4196-4ed1-b357-38f01b99b71d-kube-api-access-6nlpn\") pod \"network-check-target-l4pbv\" (UID: \"6cd2ab07-4196-4ed1-b357-38f01b99b71d\") " pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:25.559641 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:25.559625 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:28:25.559693 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:25.559646 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:28:25.559693 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:25.559655 2572 projected.go:196] Error preparing data for projected volume kube-api-access-6nlpn for pod openshift-network-diagnostics/network-check-target-l4pbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:25.559762 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:25.559697 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6cd2ab07-4196-4ed1-b357-38f01b99b71d-kube-api-access-6nlpn podName:6cd2ab07-4196-4ed1-b357-38f01b99b71d nodeName:}" failed. No retries permitted until 2026-11-03 20:28:41.559685896 +0000 UTC m=+34.356785426 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6nlpn" (UniqueName: "kubernetes.io/projected/6cd2ab07-4196-4ed1-b357-38f01b99b71d-kube-api-access-6nlpn") pod "network-check-target-l4pbv" (UID: "6cd2ab07-4196-4ed1-b357-38f01b99b71d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:25.831130 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:25.831103 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:25.831130 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:25.831125 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:25.831378 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:25.831104 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:25.831378 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:25.831206 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f28pp" podUID="2281c3da-747c-4a37-89ec-b764d8a95405" Nov 03 20:28:25.831378 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:25.831276 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sjv9" podUID="ef30d813-b7ed-417d-a9fe-85118bf77a95" Nov 03 20:28:25.831378 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:25.831356 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4pbv" podUID="6cd2ab07-4196-4ed1-b357-38f01b99b71d" Nov 03 20:28:25.959526 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:25.959366 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82vml" event={"ID":"d48c0b65-db57-43b2-8e7c-6e9e8feddf89","Type":"ContainerStarted","Data":"7206f372b92b7d5de62e71418487169fa32e71dc26921d5bc1449b52162f340a"} Nov 03 20:28:26.964938 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:26.964761 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 20:28:26.965447 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:26.965322 2572 generic.go:356] "Generic (PLEG): container finished" podID="d48c0b65-db57-43b2-8e7c-6e9e8feddf89" containerID="3dc8c3593cdfb1a0bf12dd431bd4ec5b9a97d901e47b7bd8ca0ce2b8112d3102" exitCode=1 Nov 03 20:28:26.965447 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:26.965394 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82vml" event={"ID":"d48c0b65-db57-43b2-8e7c-6e9e8feddf89","Type":"ContainerStarted","Data":"3a8c5eea8b307ac1b3caf35731a644c91bcb478096e3606b62b355d51180dff6"} Nov 03 20:28:26.965565 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:26.965529 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82vml" event={"ID":"d48c0b65-db57-43b2-8e7c-6e9e8feddf89","Type":"ContainerStarted","Data":"ee8881e06d3a398c145a00e7853562955a15074907a4c48e37ad3d3dafd50ec5"} Nov 03 20:28:26.965598 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:26.965580 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82vml" event={"ID":"d48c0b65-db57-43b2-8e7c-6e9e8feddf89","Type":"ContainerStarted","Data":"67f46244ee93b5927769f60a676d2e8a4ec53b2e5f1fef585197aaf0fe9bc924"} Nov 03 20:28:26.965643 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:26.965630 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82vml" event={"ID":"d48c0b65-db57-43b2-8e7c-6e9e8feddf89","Type":"ContainerStarted","Data":"7fe9cd1345c34fa657e2553ed0c12c6e3f2bddba6ff75c6b8d8c4bead1b84404"} Nov 03 20:28:26.965689 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:26.965676 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82vml" event={"ID":"d48c0b65-db57-43b2-8e7c-6e9e8feddf89","Type":"ContainerDied","Data":"3dc8c3593cdfb1a0bf12dd431bd4ec5b9a97d901e47b7bd8ca0ce2b8112d3102"} Nov 03 20:28:27.832416 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:27.832385 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:27.832598 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:27.832469 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sjv9" podUID="ef30d813-b7ed-417d-a9fe-85118bf77a95" Nov 03 20:28:27.832598 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:27.832479 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:27.832598 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:27.832570 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4pbv" podUID="6cd2ab07-4196-4ed1-b357-38f01b99b71d" Nov 03 20:28:27.832598 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:27.832596 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:27.832777 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:27.832646 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f28pp" podUID="2281c3da-747c-4a37-89ec-b764d8a95405" Nov 03 20:28:27.969305 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:27.969269 2572 generic.go:356] "Generic (PLEG): container finished" podID="9844c086-efc5-46f3-9d37-34c2fa013f34" containerID="ed25415b299e79f7d1814992764e8f11bb63ba3d2245bbf6f036877f7aeeac48" exitCode=0 Nov 03 20:28:27.969305 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:27.969305 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tl7xc" event={"ID":"9844c086-efc5-46f3-9d37-34c2fa013f34","Type":"ContainerDied","Data":"ed25415b299e79f7d1814992764e8f11bb63ba3d2245bbf6f036877f7aeeac48"} Nov 03 20:28:28.973988 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:28.973953 2572 generic.go:356] "Generic (PLEG): container finished" podID="9844c086-efc5-46f3-9d37-34c2fa013f34" containerID="9d723a4cbb2ec2ff9d6f41b1d7223a232a89682015e31006a3e04aef8a852958" exitCode=0 Nov 03 20:28:28.974441 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:28.974039 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tl7xc" event={"ID":"9844c086-efc5-46f3-9d37-34c2fa013f34","Type":"ContainerDied","Data":"9d723a4cbb2ec2ff9d6f41b1d7223a232a89682015e31006a3e04aef8a852958"} Nov 03 20:28:28.977108 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:28.977093 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 20:28:28.977534 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:28.977505 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82vml" event={"ID":"d48c0b65-db57-43b2-8e7c-6e9e8feddf89","Type":"ContainerStarted","Data":"68c2c0136c7bd1dcc3ca48f8d4a759195aa5f77b89c1daa6ee0a8ce3a3c206cb"} Nov 03 20:28:29.831346 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:29.831306 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:29.831591 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:29.831355 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:29.831591 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:29.831431 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4pbv" podUID="6cd2ab07-4196-4ed1-b357-38f01b99b71d" Nov 03 20:28:29.831591 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:29.831514 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f28pp" podUID="2281c3da-747c-4a37-89ec-b764d8a95405" Nov 03 20:28:29.831591 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:29.831550 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:29.831741 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:29.831648 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sjv9" podUID="ef30d813-b7ed-417d-a9fe-85118bf77a95" Nov 03 20:28:29.982650 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:29.982624 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tl7xc" event={"ID":"9844c086-efc5-46f3-9d37-34c2fa013f34","Type":"ContainerStarted","Data":"b094277bacdde7a26eaab0b9a4d948cf65a0dfa0622e898d51f95bffca522f14"} Nov 03 20:28:30.118252 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:30.118157 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tl7xc" podStartSLOduration=8.719498854 podStartE2EDuration="22.118144904s" totalImagesPullingTime="13.39864605s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:30.117770756 +0000 UTC m=+22.914870306" watchObservedRunningTime="2026-11-03 20:28:30.118144904 +0000 UTC m=+22.915244453" Nov 03 20:28:30.987807 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:30.987779 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 20:28:30.988497 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:30.988152 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82vml" event={"ID":"d48c0b65-db57-43b2-8e7c-6e9e8feddf89","Type":"ContainerStarted","Data":"a6c50eedab8fbba40af27973a63a17e95a025df6c03d2533592777d5e8085aae"} Nov 03 20:28:30.989803 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:30.989670 2572 scope.go:122] "RemoveContainer" containerID="3dc8c3593cdfb1a0bf12dd431bd4ec5b9a97d901e47b7bd8ca0ce2b8112d3102" Nov 03 20:28:31.831373 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:31.831165 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:31.831533 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:31.831232 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:31.831533 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:31.831461 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sjv9" podUID="ef30d813-b7ed-417d-a9fe-85118bf77a95" Nov 03 20:28:31.831611 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:31.831535 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f28pp" podUID="2281c3da-747c-4a37-89ec-b764d8a95405" Nov 03 20:28:31.831611 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:31.831271 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:31.831669 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:31.831614 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4pbv" podUID="6cd2ab07-4196-4ed1-b357-38f01b99b71d" Nov 03 20:28:31.994180 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:31.994150 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 20:28:31.994580 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:31.994557 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82vml" event={"ID":"d48c0b65-db57-43b2-8e7c-6e9e8feddf89","Type":"ContainerStarted","Data":"17ff98404167a1e7461947827da7606ca8904d0321597929e6d067508a109da3"} Nov 03 20:28:31.995295 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:31.994875 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:31.995295 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:31.994955 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:31.995295 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:31.995002 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:32.011610 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:32.011570 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:32.012824 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:32.012808 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:32.168921 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:32.168839 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-82vml" podStartSLOduration=8.363583234 podStartE2EDuration="24.168825901s" totalImagesPullingTime="15.805242667s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:32.168599404 +0000 UTC m=+24.965698954" watchObservedRunningTime="2026-11-03 20:28:32.168825901 +0000 UTC m=+24.965925444" Nov 03 20:28:33.219587 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:33.219545 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f28pp"] Nov 03 20:28:33.220026 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:33.219698 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:33.220026 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:33.219788 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f28pp" podUID="2281c3da-747c-4a37-89ec-b764d8a95405" Nov 03 20:28:33.220026 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:33.219998 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2sjv9"] Nov 03 20:28:33.220142 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:33.220074 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:33.220179 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:33.220137 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sjv9" podUID="ef30d813-b7ed-417d-a9fe-85118bf77a95" Nov 03 20:28:33.221316 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:33.221300 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l4pbv"] Nov 03 20:28:33.221394 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:33.221384 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:33.221472 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:33.221455 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4pbv" podUID="6cd2ab07-4196-4ed1-b357-38f01b99b71d" Nov 03 20:28:33.276967 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:33.276944 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ccxwr" Nov 03 20:28:33.277082 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:33.277025 2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:28:33.277475 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:33.277463 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ccxwr" Nov 03 20:28:34.831152 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:34.831090 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:34.831531 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:34.831090 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:34.831531 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:34.831210 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f28pp" podUID="2281c3da-747c-4a37-89ec-b764d8a95405" Nov 03 20:28:34.831531 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:34.831090 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:34.831531 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:34.831270 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sjv9" podUID="ef30d813-b7ed-417d-a9fe-85118bf77a95" Nov 03 20:28:34.831531 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:34.831361 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4pbv" podUID="6cd2ab07-4196-4ed1-b357-38f01b99b71d" Nov 03 20:28:36.830902 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:36.830868 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:36.831261 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:36.830885 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:36.831261 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:36.831030 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:36.831261 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:36.831034 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sjv9" podUID="ef30d813-b7ed-417d-a9fe-85118bf77a95" Nov 03 20:28:36.831261 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:36.831100 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4pbv" podUID="6cd2ab07-4196-4ed1-b357-38f01b99b71d" Nov 03 20:28:36.831261 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:36.831153 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f28pp" podUID="2281c3da-747c-4a37-89ec-b764d8a95405" Nov 03 20:28:38.486776 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.486738 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeReady" Nov 03 20:28:38.487167 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.486860 2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 20:28:38.599997 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.599971 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jjqxd"] Nov 03 20:28:38.618236 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.618208 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:38.650856 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.650833 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5jfs9"] Nov 03 20:28:38.663318 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.663295 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5jfs9" Nov 03 20:28:38.664652 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.664634 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jjqxd"] Nov 03 20:28:38.671344 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.671315 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 20:28:38.671458 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.671328 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 20:28:38.671572 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.671556 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pqh8h\"" Nov 03 20:28:38.701776 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.701755 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5jfs9"] Nov 03 20:28:38.757479 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.757414 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:38.757479 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.757448 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/480d6b9d-4f58-4779-a2cb-aa4b64f93881-config-volume\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:38.757479 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.757471 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/480d6b9d-4f58-4779-a2cb-aa4b64f93881-tmp-dir\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:38.757666 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.757487 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m49g\" (UniqueName: \"kubernetes.io/projected/480d6b9d-4f58-4779-a2cb-aa4b64f93881-kube-api-access-2m49g\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:38.757666 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.757620 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6ls8\" (UniqueName: \"kubernetes.io/projected/7e65f973-499a-4dac-af00-b485fb5dbbfe-kube-api-access-c6ls8\") pod \"ingress-canary-5jfs9\" (UID: \"7e65f973-499a-4dac-af00-b485fb5dbbfe\") " pod="openshift-ingress-canary/ingress-canary-5jfs9" Nov 03 20:28:38.757666 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.757655 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e65f973-499a-4dac-af00-b485fb5dbbfe-cert\") pod \"ingress-canary-5jfs9\" (UID: \"7e65f973-499a-4dac-af00-b485fb5dbbfe\") " pod="openshift-ingress-canary/ingress-canary-5jfs9" Nov 03 20:28:38.771678 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.771660 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 20:28:38.771769 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.771734 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 20:28:38.771866 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.771853 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 20:28:38.771923 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.771911 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-fpmzx\"" Nov 03 20:28:38.831207 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.831180 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:38.831278 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.831189 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:38.831322 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.831295 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:38.858200 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.858179 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6ls8\" (UniqueName: \"kubernetes.io/projected/7e65f973-499a-4dac-af00-b485fb5dbbfe-kube-api-access-c6ls8\") pod \"ingress-canary-5jfs9\" (UID: \"7e65f973-499a-4dac-af00-b485fb5dbbfe\") " pod="openshift-ingress-canary/ingress-canary-5jfs9" Nov 03 20:28:38.858300 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.858214 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e65f973-499a-4dac-af00-b485fb5dbbfe-cert\") pod \"ingress-canary-5jfs9\" (UID: \"7e65f973-499a-4dac-af00-b485fb5dbbfe\") " pod="openshift-ingress-canary/ingress-canary-5jfs9" Nov 03 20:28:38.858300 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.858259 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:38.858300 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.858280 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/480d6b9d-4f58-4779-a2cb-aa4b64f93881-config-volume\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:38.858467 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.858308 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/480d6b9d-4f58-4779-a2cb-aa4b64f93881-tmp-dir\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:38.858467 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.858347 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m49g\" (UniqueName: \"kubernetes.io/projected/480d6b9d-4f58-4779-a2cb-aa4b64f93881-kube-api-access-2m49g\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:38.858467 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:38.858377 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:28:38.858467 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:38.858443 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls podName:480d6b9d-4f58-4779-a2cb-aa4b64f93881 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:39.358425739 +0000 UTC m=+32.155525282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls") pod "dns-default-jjqxd" (UID: "480d6b9d-4f58-4779-a2cb-aa4b64f93881") : secret "dns-default-metrics-tls" not found Nov 03 20:28:38.858778 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.858754 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/480d6b9d-4f58-4779-a2cb-aa4b64f93881-tmp-dir\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:38.858946 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.858926 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/480d6b9d-4f58-4779-a2cb-aa4b64f93881-config-volume\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:38.862419 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.862282 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e65f973-499a-4dac-af00-b485fb5dbbfe-cert\") pod \"ingress-canary-5jfs9\" (UID: \"7e65f973-499a-4dac-af00-b485fb5dbbfe\") " pod="openshift-ingress-canary/ingress-canary-5jfs9" Nov 03 20:28:38.884665 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.884642 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 20:28:38.884768 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.884753 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 20:28:38.884835 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.884820 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 20:28:38.884995 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.884981 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jxjwz\"" Nov 03 20:28:38.885035 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.885002 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kpkvl\"" Nov 03 20:28:38.885035 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.885021 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h9ssl\"" Nov 03 20:28:38.885111 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.885067 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 20:28:38.920218 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.920198 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m49g\" (UniqueName: \"kubernetes.io/projected/480d6b9d-4f58-4779-a2cb-aa4b64f93881-kube-api-access-2m49g\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:38.920299 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.920251 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6ls8\" (UniqueName: \"kubernetes.io/projected/7e65f973-499a-4dac-af00-b485fb5dbbfe-kube-api-access-c6ls8\") pod \"ingress-canary-5jfs9\" (UID: \"7e65f973-499a-4dac-af00-b485fb5dbbfe\") " pod="openshift-ingress-canary/ingress-canary-5jfs9" Nov 03 20:28:38.986495 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:38.986467 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5jfs9" Nov 03 20:28:39.362501 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:39.362462 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:39.362688 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:39.362627 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:28:39.362688 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:39.362680 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls podName:480d6b9d-4f58-4779-a2cb-aa4b64f93881 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:40.362667585 +0000 UTC m=+33.159767113 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls") pod "dns-default-jjqxd" (UID: "480d6b9d-4f58-4779-a2cb-aa4b64f93881") : secret "dns-default-metrics-tls" not found Nov 03 20:28:39.421923 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:39.421828 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5jfs9"] Nov 03 20:28:39.425438 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:39.425402 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e65f973_499a_4dac_af00_b485fb5dbbfe.slice/crio-444bb3bce81beb9241034c1e8c8a8e76d27752a99c32addfbcc916c53563a2e1 WatchSource:0}: Error finding container 444bb3bce81beb9241034c1e8c8a8e76d27752a99c32addfbcc916c53563a2e1: Status 404 returned error can't find the container with id 444bb3bce81beb9241034c1e8c8a8e76d27752a99c32addfbcc916c53563a2e1 Nov 03 20:28:40.016156 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:40.016125 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5jfs9" event={"ID":"7e65f973-499a-4dac-af00-b485fb5dbbfe","Type":"ContainerStarted","Data":"444bb3bce81beb9241034c1e8c8a8e76d27752a99c32addfbcc916c53563a2e1"} Nov 03 20:28:40.368913 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:40.368866 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:40.369086 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:40.369054 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:28:40.369141 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:40.369128 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls podName:480d6b9d-4f58-4779-a2cb-aa4b64f93881 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:42.369108671 +0000 UTC m=+35.166208221 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls") pod "dns-default-jjqxd" (UID: "480d6b9d-4f58-4779-a2cb-aa4b64f93881") : secret "dns-default-metrics-tls" not found Nov 03 20:28:41.478141 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:41.478109 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs\") pod \"network-metrics-daemon-f28pp\" (UID: \"2281c3da-747c-4a37-89ec-b764d8a95405\") " pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:41.478613 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:41.478167 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef30d813-b7ed-417d-a9fe-85118bf77a95-original-pull-secret\") pod \"global-pull-secret-syncer-2sjv9\" (UID: \"ef30d813-b7ed-417d-a9fe-85118bf77a95\") " pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:41.478807 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:41.478781 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 20:28:41.478977 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:41.478865 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs podName:2281c3da-747c-4a37-89ec-b764d8a95405 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:13.478844567 +0000 UTC m=+66.275944114 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs") pod "network-metrics-daemon-f28pp" (UID: "2281c3da-747c-4a37-89ec-b764d8a95405") : secret "metrics-daemon-secret" not found Nov 03 20:28:41.481283 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:41.481254 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef30d813-b7ed-417d-a9fe-85118bf77a95-original-pull-secret\") pod \"global-pull-secret-syncer-2sjv9\" (UID: \"ef30d813-b7ed-417d-a9fe-85118bf77a95\") " pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:41.578846 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:41.578805 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nlpn\" (UniqueName: \"kubernetes.io/projected/6cd2ab07-4196-4ed1-b357-38f01b99b71d-kube-api-access-6nlpn\") pod \"network-check-target-l4pbv\" (UID: \"6cd2ab07-4196-4ed1-b357-38f01b99b71d\") " pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:41.581786 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:41.581761 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nlpn\" (UniqueName: \"kubernetes.io/projected/6cd2ab07-4196-4ed1-b357-38f01b99b71d-kube-api-access-6nlpn\") pod \"network-check-target-l4pbv\" (UID: \"6cd2ab07-4196-4ed1-b357-38f01b99b71d\") " pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:41.597503 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:41.597476 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:41.611649 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:41.611625 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:42.001900 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.001867 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2sjv9"] Nov 03 20:28:42.006445 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:42.006414 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef30d813_b7ed_417d_a9fe_85118bf77a95.slice/crio-f62cdd8d3dd5a0439afe7c3e7cb27b1c794287bb49d5dd8f9deb51158f2a315b WatchSource:0}: Error finding container f62cdd8d3dd5a0439afe7c3e7cb27b1c794287bb49d5dd8f9deb51158f2a315b: Status 404 returned error can't find the container with id f62cdd8d3dd5a0439afe7c3e7cb27b1c794287bb49d5dd8f9deb51158f2a315b Nov 03 20:28:42.024103 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.024068 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2sjv9" event={"ID":"ef30d813-b7ed-417d-a9fe-85118bf77a95","Type":"ContainerStarted","Data":"f62cdd8d3dd5a0439afe7c3e7cb27b1c794287bb49d5dd8f9deb51158f2a315b"} Nov 03 20:28:42.052171 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.052139 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l4pbv"] Nov 03 20:28:42.056696 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:42.056672 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd2ab07_4196_4ed1_b357_38f01b99b71d.slice/crio-916ecd53f7745d6f5bc1f3c7ab629fc90d57467bc749b5a76379d7ecedfcadc1 WatchSource:0}: Error finding container 916ecd53f7745d6f5bc1f3c7ab629fc90d57467bc749b5a76379d7ecedfcadc1: Status 404 returned error can't find the container with id 916ecd53f7745d6f5bc1f3c7ab629fc90d57467bc749b5a76379d7ecedfcadc1 Nov 03 20:28:42.369431 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.369397 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-67nxb"] Nov 03 20:28:42.384272 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.384250 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:42.384792 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.384766 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:42.384932 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:42.384917 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:28:42.384999 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:42.384989 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls podName:480d6b9d-4f58-4779-a2cb-aa4b64f93881 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:46.384971838 +0000 UTC m=+39.182071385 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls") pod "dns-default-jjqxd" (UID: "480d6b9d-4f58-4779-a2cb-aa4b64f93881") : secret "dns-default-metrics-tls" not found Nov 03 20:28:42.426140 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.426113 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-67nxb"] Nov 03 20:28:42.437863 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.437826 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 20:28:42.437863 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.437846 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 20:28:42.437863 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.437852 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 20:28:42.438044 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.437902 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 20:28:42.476396 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.476369 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ntq9n\"" Nov 03 20:28:42.485926 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.485898 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b013faae-082b-495b-9d77-b4371e82e379-crio-socket\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:42.486275 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.485968 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b013faae-082b-495b-9d77-b4371e82e379-data-volume\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:42.486275 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.485993 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b013faae-082b-495b-9d77-b4371e82e379-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:42.486275 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.486158 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szxvh\" (UniqueName: \"kubernetes.io/projected/b013faae-082b-495b-9d77-b4371e82e379-kube-api-access-szxvh\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:42.486275 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.486211 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:42.586716 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.586681 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szxvh\" (UniqueName: \"kubernetes.io/projected/b013faae-082b-495b-9d77-b4371e82e379-kube-api-access-szxvh\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:42.586841 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.586727 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:42.586841 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.586747 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b013faae-082b-495b-9d77-b4371e82e379-crio-socket\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:42.586841 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.586777 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b013faae-082b-495b-9d77-b4371e82e379-data-volume\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:42.586951 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:42.586852 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:28:42.586951 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.586893 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b013faae-082b-495b-9d77-b4371e82e379-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:42.586951 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:42.586921 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls podName:b013faae-082b-495b-9d77-b4371e82e379 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:43.086902836 +0000 UTC m=+35.884002384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls") pod "insights-runtime-extractor-67nxb" (UID: "b013faae-082b-495b-9d77-b4371e82e379") : secret "insights-runtime-extractor-tls" not found Nov 03 20:28:42.587100 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:42.586951 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 20:28:42.587100 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.586997 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b013faae-082b-495b-9d77-b4371e82e379-crio-socket\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:42.587100 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:42.587049 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b013faae-082b-495b-9d77-b4371e82e379-kube-rbac-proxy-cm podName:b013faae-082b-495b-9d77-b4371e82e379 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:43.087030814 +0000 UTC m=+35.884130355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b013faae-082b-495b-9d77-b4371e82e379-kube-rbac-proxy-cm") pod "insights-runtime-extractor-67nxb" (UID: "b013faae-082b-495b-9d77-b4371e82e379") : configmap "kube-rbac-proxy" not found Nov 03 20:28:42.587100 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.587080 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b013faae-082b-495b-9d77-b4371e82e379-data-volume\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:42.608915 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.608881 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-rr8h2-bjl4n"] Nov 03 20:28:42.620978 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.620919 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" Nov 03 20:28:42.645268 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.645249 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szxvh\" (UniqueName: \"kubernetes.io/projected/b013faae-082b-495b-9d77-b4371e82e379-kube-api-access-szxvh\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:42.666010 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.665977 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rr8h2-bjl4n"] Nov 03 20:28:42.674252 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.674227 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 20:28:42.674383 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.674233 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xvpcc\"" Nov 03 20:28:42.745019 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.744997 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 20:28:42.788016 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.787986 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34b6b405-a677-4fef-8ee6-37d84a1c5902-trusted-ca-bundle\") pod \"periodic-gathering-rr8h2-bjl4n\" (UID: \"34b6b405-a677-4fef-8ee6-37d84a1c5902\") " pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" Nov 03 20:28:42.788163 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.788048 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9qrh\" (UniqueName: \"kubernetes.io/projected/34b6b405-a677-4fef-8ee6-37d84a1c5902-kube-api-access-h9qrh\") pod \"periodic-gathering-rr8h2-bjl4n\" (UID: \"34b6b405-a677-4fef-8ee6-37d84a1c5902\") " pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" Nov 03 20:28:42.788163 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.788120 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34b6b405-a677-4fef-8ee6-37d84a1c5902-service-ca-bundle\") pod \"periodic-gathering-rr8h2-bjl4n\" (UID: \"34b6b405-a677-4fef-8ee6-37d84a1c5902\") " pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" Nov 03 20:28:42.788275 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.788180 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/34b6b405-a677-4fef-8ee6-37d84a1c5902-archives-path\") pod \"periodic-gathering-rr8h2-bjl4n\" (UID: \"34b6b405-a677-4fef-8ee6-37d84a1c5902\") " pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" Nov 03 20:28:42.888611 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.888532 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9qrh\" (UniqueName: \"kubernetes.io/projected/34b6b405-a677-4fef-8ee6-37d84a1c5902-kube-api-access-h9qrh\") pod \"periodic-gathering-rr8h2-bjl4n\" (UID: \"34b6b405-a677-4fef-8ee6-37d84a1c5902\") " pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" Nov 03 20:28:42.888611 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.888568 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34b6b405-a677-4fef-8ee6-37d84a1c5902-service-ca-bundle\") pod \"periodic-gathering-rr8h2-bjl4n\" (UID: \"34b6b405-a677-4fef-8ee6-37d84a1c5902\") " pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" Nov 03 20:28:42.888611 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.888606 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/34b6b405-a677-4fef-8ee6-37d84a1c5902-archives-path\") pod \"periodic-gathering-rr8h2-bjl4n\" (UID: \"34b6b405-a677-4fef-8ee6-37d84a1c5902\") " pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" Nov 03 20:28:42.888872 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.888673 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34b6b405-a677-4fef-8ee6-37d84a1c5902-trusted-ca-bundle\") pod \"periodic-gathering-rr8h2-bjl4n\" (UID: \"34b6b405-a677-4fef-8ee6-37d84a1c5902\") " pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" Nov 03 20:28:42.889072 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.889049 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/34b6b405-a677-4fef-8ee6-37d84a1c5902-archives-path\") pod \"periodic-gathering-rr8h2-bjl4n\" (UID: \"34b6b405-a677-4fef-8ee6-37d84a1c5902\") " pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" Nov 03 20:28:42.889180 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.889156 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34b6b405-a677-4fef-8ee6-37d84a1c5902-service-ca-bundle\") pod \"periodic-gathering-rr8h2-bjl4n\" (UID: \"34b6b405-a677-4fef-8ee6-37d84a1c5902\") " pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" Nov 03 20:28:42.889460 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.889440 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34b6b405-a677-4fef-8ee6-37d84a1c5902-trusted-ca-bundle\") pod \"periodic-gathering-rr8h2-bjl4n\" (UID: \"34b6b405-a677-4fef-8ee6-37d84a1c5902\") " pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" Nov 03 20:28:42.946031 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:42.946009 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9qrh\" (UniqueName: \"kubernetes.io/projected/34b6b405-a677-4fef-8ee6-37d84a1c5902-kube-api-access-h9qrh\") pod \"periodic-gathering-rr8h2-bjl4n\" (UID: \"34b6b405-a677-4fef-8ee6-37d84a1c5902\") " pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" Nov 03 20:28:43.028629 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:43.028592 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5jfs9" event={"ID":"7e65f973-499a-4dac-af00-b485fb5dbbfe","Type":"ContainerStarted","Data":"9b0ceea38bf9456e85bc613ec5a1a7a2f83a37b42b387dcb217a0dc4ef124949"} Nov 03 20:28:43.030017 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:43.029983 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l4pbv" event={"ID":"6cd2ab07-4196-4ed1-b357-38f01b99b71d","Type":"ContainerStarted","Data":"916ecd53f7745d6f5bc1f3c7ab629fc90d57467bc749b5a76379d7ecedfcadc1"} Nov 03 20:28:43.091322 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:43.091286 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b013faae-082b-495b-9d77-b4371e82e379-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:43.091459 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:43.091388 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:43.091532 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:43.091517 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:28:43.091593 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:43.091579 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls podName:b013faae-082b-495b-9d77-b4371e82e379 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:44.091568 +0000 UTC m=+36.888667529 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls") pod "insights-runtime-extractor-67nxb" (UID: "b013faae-082b-495b-9d77-b4371e82e379") : secret "insights-runtime-extractor-tls" not found Nov 03 20:28:43.091855 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:43.091838 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 20:28:43.091926 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:43.091879 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b013faae-082b-495b-9d77-b4371e82e379-kube-rbac-proxy-cm podName:b013faae-082b-495b-9d77-b4371e82e379 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:44.091869849 +0000 UTC m=+36.888969376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b013faae-082b-495b-9d77-b4371e82e379-kube-rbac-proxy-cm") pod "insights-runtime-extractor-67nxb" (UID: "b013faae-082b-495b-9d77-b4371e82e379") : configmap "kube-rbac-proxy" not found Nov 03 20:28:43.144243 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:43.144155 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5jfs9" podStartSLOduration=1.930202161 podStartE2EDuration="5.144137457s" totalImagesPullingTime="3.213935296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:43.143974618 +0000 UTC m=+35.941074170" watchObservedRunningTime="2026-11-03 20:28:43.144137457 +0000 UTC m=+35.941237006" Nov 03 20:28:43.243565 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:43.243520 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" Nov 03 20:28:43.432433 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:43.431446 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gj9nc"] Nov 03 20:28:43.449521 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:43.449490 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gj9nc" Nov 03 20:28:43.487641 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:43.487409 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gj9nc"] Nov 03 20:28:43.578707 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:43.578120 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 20:28:43.578707 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:43.578199 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 20:28:43.578707 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:43.578322 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8hb94\"" Nov 03 20:28:43.595789 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:43.595761 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwv4c\" (UniqueName: \"kubernetes.io/projected/3514724f-deea-4b4e-bce9-48ce7862d6f0-kube-api-access-hwv4c\") pod \"migrator-66df5d496b-gj9nc\" (UID: \"3514724f-deea-4b4e-bce9-48ce7862d6f0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gj9nc" Nov 03 20:28:43.678500 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:43.677932 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rr8h2-bjl4n"] Nov 03 20:28:43.696463 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:43.696431 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwv4c\" (UniqueName: \"kubernetes.io/projected/3514724f-deea-4b4e-bce9-48ce7862d6f0-kube-api-access-hwv4c\") pod \"migrator-66df5d496b-gj9nc\" (UID: \"3514724f-deea-4b4e-bce9-48ce7862d6f0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gj9nc" Nov 03 20:28:43.758803 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:43.758774 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwv4c\" (UniqueName: \"kubernetes.io/projected/3514724f-deea-4b4e-bce9-48ce7862d6f0-kube-api-access-hwv4c\") pod \"migrator-66df5d496b-gj9nc\" (UID: \"3514724f-deea-4b4e-bce9-48ce7862d6f0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gj9nc" Nov 03 20:28:43.774206 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:43.773566 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gj9nc" Nov 03 20:28:44.034942 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:44.034850 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" event={"ID":"34b6b405-a677-4fef-8ee6-37d84a1c5902","Type":"ContainerStarted","Data":"14f0e7044649ef39b41f8df4f11301f5b107fb03376661ebd8b18605a9953596"} Nov 03 20:28:44.100863 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:44.100785 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:44.101041 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:44.100871 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b013faae-082b-495b-9d77-b4371e82e379-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:44.101041 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:44.100965 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 20:28:44.101041 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:44.101035 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b013faae-082b-495b-9d77-b4371e82e379-kube-rbac-proxy-cm podName:b013faae-082b-495b-9d77-b4371e82e379 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:46.101018507 +0000 UTC m=+38.898118042 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b013faae-082b-495b-9d77-b4371e82e379-kube-rbac-proxy-cm") pod "insights-runtime-extractor-67nxb" (UID: "b013faae-082b-495b-9d77-b4371e82e379") : configmap "kube-rbac-proxy" not found Nov 03 20:28:44.101325 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:44.101300 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:28:44.101454 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:44.101402 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls podName:b013faae-082b-495b-9d77-b4371e82e379 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:46.101384803 +0000 UTC m=+38.898484347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls") pod "insights-runtime-extractor-67nxb" (UID: "b013faae-082b-495b-9d77-b4371e82e379") : secret "insights-runtime-extractor-tls" not found Nov 03 20:28:44.177180 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:44.176549 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gj9nc"] Nov 03 20:28:44.181569 ip-10-0-138-26 kubenswrapper[2572]: W1103 20:28:44.181533 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3514724f_deea_4b4e_bce9_48ce7862d6f0.slice/crio-8426b10369feb5597e03c517e9e3d9a6a745ea14724782cf2b4b6bb792b1c120 WatchSource:0}: Error finding container 8426b10369feb5597e03c517e9e3d9a6a745ea14724782cf2b4b6bb792b1c120: Status 404 returned error can't find the container with id 8426b10369feb5597e03c517e9e3d9a6a745ea14724782cf2b4b6bb792b1c120 Nov 03 20:28:45.040887 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:45.040811 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gj9nc" event={"ID":"3514724f-deea-4b4e-bce9-48ce7862d6f0","Type":"ContainerStarted","Data":"8426b10369feb5597e03c517e9e3d9a6a745ea14724782cf2b4b6bb792b1c120"} Nov 03 20:28:46.116777 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:46.116742 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:46.117389 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:46.116805 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b013faae-082b-495b-9d77-b4371e82e379-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:46.117389 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:46.116900 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 20:28:46.117389 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:46.116953 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b013faae-082b-495b-9d77-b4371e82e379-kube-rbac-proxy-cm podName:b013faae-082b-495b-9d77-b4371e82e379 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:50.116939305 +0000 UTC m=+42.914038833 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b013faae-082b-495b-9d77-b4371e82e379-kube-rbac-proxy-cm") pod "insights-runtime-extractor-67nxb" (UID: "b013faae-082b-495b-9d77-b4371e82e379") : configmap "kube-rbac-proxy" not found Nov 03 20:28:46.117389 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:46.117307 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:28:46.117389 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:46.117377 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls podName:b013faae-082b-495b-9d77-b4371e82e379 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:50.117363497 +0000 UTC m=+42.914463028 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls") pod "insights-runtime-extractor-67nxb" (UID: "b013faae-082b-495b-9d77-b4371e82e379") : secret "insights-runtime-extractor-tls" not found Nov 03 20:28:46.419130 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:46.419041 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:46.419308 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:46.419224 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:28:46.419412 ip-10-0-138-26 kubenswrapper[2572]: E1103 20:28:46.419350 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls podName:480d6b9d-4f58-4779-a2cb-aa4b64f93881 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:54.419307426 +0000 UTC m=+47.216406959 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls") pod "dns-default-jjqxd" (UID: "480d6b9d-4f58-4779-a2cb-aa4b64f93881") : secret "dns-default-metrics-tls" not found Nov 03 20:28:47.049648 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:47.049611 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gj9nc" event={"ID":"3514724f-deea-4b4e-bce9-48ce7862d6f0","Type":"ContainerStarted","Data":"1f89edda4f08acb01fe72f72e12a24026cd2cc07d8fee6b49f550c06c259b0ae"} Nov 03 20:28:47.051353 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:47.051312 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l4pbv" event={"ID":"6cd2ab07-4196-4ed1-b357-38f01b99b71d","Type":"ContainerStarted","Data":"fe9827ec25386586b5e835dda69962a0f86459f6b811ea2e34126d99348c1a3d"} Nov 03 20:28:47.051508 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:47.051490 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:47.303744 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:47.303687 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-l4pbv" podStartSLOduration=34.544191221 podStartE2EDuration="39.303672584s" totalImagesPullingTime="4.759481363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:47.303419164 +0000 UTC m=+40.100518716" watchObservedRunningTime="2026-11-03 20:28:47.303672584 +0000 UTC m=+40.100772133" Nov 03 20:28:47.304133 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:47.304005 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gj9nc" podStartSLOduration=1.667284669 podStartE2EDuration="4.303999527s" totalImagesPullingTime="2.636714858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:47.184053749 +0000 UTC m=+39.981153303" watchObservedRunningTime="2026-11-03 20:28:47.303999527 +0000 UTC m=+40.101099077" Nov 03 20:28:48.055367 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:48.055310 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" event={"ID":"34b6b405-a677-4fef-8ee6-37d84a1c5902","Type":"ContainerStarted","Data":"4dca622fd78277fac0622261e66b08c483bbc760b969cd1b39386d5165d46c20"} Nov 03 20:28:48.171740 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:48.171691 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" podStartSLOduration=2.728758864 podStartE2EDuration="6.171676787s" totalImagesPullingTime="3.442917923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:48.170814998 +0000 UTC m=+40.967914563" watchObservedRunningTime="2026-11-03 20:28:48.171676787 +0000 UTC m=+40.968776337" Nov 03 20:28:48.355504 ip-10-0-138-26 kubenswrapper[2572]: I1103 20:28:48.355413 2572 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:28:48.355567 ip-10-0-138-26 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 20:28:48.365714 ip-10-0-138-26 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 20:28:48.365875 ip-10-0-138-26 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 20:28:48.365942 ip-10-0-138-26 systemd[1]: kubelet.service: Consumed 2.369s CPU time. Nov 03 20:28:48.380111 ip-10-0-138-26 systemd[1]: Starting Kubernetes Kubelet... Nov 03 20:28:48.479107 ip-10-0-138-26 kubenswrapper[4493]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:28:48.479107 ip-10-0-138-26 kubenswrapper[4493]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 20:28:48.479107 ip-10-0-138-26 kubenswrapper[4493]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:28:48.479107 ip-10-0-138-26 kubenswrapper[4493]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:28:48.482348 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482244 4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:48.482459 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482358 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:48.482459 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482366 4493 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:48.482459 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482370 4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:48.482459 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482373 4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:48.482459 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482377 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:48.482459 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482380 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:48.482459 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482383 4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:48.482459 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482385 4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:48.482459 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482388 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:48.482459 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482393 4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:48.482459 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482397 4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:48.482459 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482401 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:48.482459 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482403 4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:48.482459 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482406 4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:48.482459 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482409 4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:48.482459 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482412 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:48.482459 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482416 4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:48.482459 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482419 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:48.482459 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482430 4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:48.482978 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482433 4493 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:48.482978 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482436 4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:48.482978 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482438 4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:48.482978 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482441 4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:48.482978 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482444 4493 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:48.482978 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482448 4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:48.482978 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482451 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:48.482978 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482455 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:48.482978 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482458 4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:48.482978 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482461 4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:48.482978 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482464 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:48.482978 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482466 4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:48.482978 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482469 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:48.482978 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482472 4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:48.482978 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482475 4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:48.482978 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482478 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:48.482978 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482481 4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:48.482978 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482483 4493 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:48.482978 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482486 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:48.482978 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482488 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:48.483496 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482491 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:48.483496 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482494 4493 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:48.483496 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482496 4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:48.483496 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482499 4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:48.483496 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482501 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:48.483496 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482504 4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:48.483496 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482507 4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:48.483496 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482510 4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:48.483496 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482514 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:48.483496 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482517 4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:48.483496 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482521 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:48.483496 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482524 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:48.483496 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482527 4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:48.483496 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482530 4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:48.483496 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482532 4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:48.483496 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482535 4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:48.483496 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482537 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:48.483496 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482540 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:48.483496 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482543 4493 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:48.483981 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482545 4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:48.483981 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482548 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:48.483981 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482550 4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:48.483981 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482553 4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:48.483981 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482556 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:48.483981 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482558 4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:48.483981 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482561 4493 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:48.483981 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482563 4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:48.483981 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482566 4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:48.483981 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482568 4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:48.483981 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482571 4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:48.483981 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482573 4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:48.483981 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482576 4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:48.483981 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482578 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:48.483981 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482581 4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:48.483981 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482583 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:48.483981 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482586 4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:48.483981 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482589 4493 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:48.483981 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482591 4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:48.484478 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482594 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:48.484478 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482597 4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:48.484478 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482600 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:48.484478 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482603 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:48.484478 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482605 4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:48.484478 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482608 4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:48.484478 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482612 4493 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:48.484478 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482614 4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:48.484478 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482617 4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:48.484478 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482620 4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:48.484478 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482623 4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:48.484478 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482625 4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:48.484478 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482628 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:48.484478 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482631 4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:48.484478 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482633 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:48.484478 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482636 4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:48.484478 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482638 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:48.484478 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482641 4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:48.484478 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482644 4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:48.484959 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482646 4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:48.484959 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482648 4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:48.484959 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482651 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:48.484959 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482654 4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:48.484959 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482657 4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:48.484959 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482660 4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:48.484959 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482663 4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:48.484959 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482666 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:48.484959 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.482668 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:48.484959 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483419 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:48.484959 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483426 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:48.484959 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483429 4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:48.484959 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483432 4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:48.484959 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483435 4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:48.484959 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483438 4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:48.484959 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483441 4493 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:48.484959 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483443 4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:48.484959 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483446 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:48.484959 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483449 4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:48.485447 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483453 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:48.485447 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483456 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:48.485447 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483459 4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:48.485447 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483462 4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:48.485447 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483465 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:48.485447 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483467 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:48.485447 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483470 4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:48.485447 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483473 4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:48.485447 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483476 4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:48.485447 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483478 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:48.485447 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483481 4493 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:48.485447 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483484 4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:48.485447 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483486 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:48.485447 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483489 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:48.485447 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483491 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:48.485447 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483494 4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:48.485447 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483496 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:48.485447 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483499 4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:48.485447 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483502 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:48.485916 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483505 4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:48.485916 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483507 4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:48.485916 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483510 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:48.485916 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483512 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:48.485916 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483515 4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:48.485916 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483518 4493 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:48.485916 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483521 4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:48.485916 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483524 4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:48.485916 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483526 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:48.485916 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483529 4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:48.485916 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483534 4493 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:48.485916 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483536 4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:48.485916 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483547 4493 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:48.485916 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483550 4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:48.485916 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483553 4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:48.485916 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483557 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:48.485916 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483559 4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:48.485916 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483562 4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:48.485916 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483565 4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:48.485916 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483567 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:48.486436 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483570 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:48.486436 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483572 4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:48.486436 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483575 4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:48.486436 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483577 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:48.486436 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483580 4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:48.486436 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483583 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:48.486436 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483585 4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:48.486436 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483588 4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:48.486436 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483590 4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:48.486436 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483594 4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:48.486436 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483598 4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:48.486436 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483601 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:48.486436 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483604 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:48.486436 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483606 4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:48.486436 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483611 4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:48.486436 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483615 4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:48.486436 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483618 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:48.486436 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483621 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:48.486436 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483624 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:48.486957 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483626 4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:48.486957 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483629 4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:48.486957 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483632 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:48.486957 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483635 4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:48.486957 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483638 4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:48.486957 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483641 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:48.486957 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483644 4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:48.486957 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483646 4493 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:48.486957 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483650 4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:48.486957 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483652 4493 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:48.486957 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483655 4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:48.486957 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483657 4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:48.486957 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483660 4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:48.486957 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483662 4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:48.486957 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483665 4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:48.486957 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483667 4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:48.486957 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483670 4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:48.486957 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483672 4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:48.486957 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483675 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:48.486957 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483677 4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:48.487450 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483680 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:48.487450 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483683 4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:48.487450 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483685 4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:48.487450 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483688 4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:48.487450 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483691 4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:48.487450 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483694 4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:48.487450 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483696 4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:48.487450 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483700 4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:48.487450 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483702 4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:48.487450 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483705 4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:48.487450 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483708 4493 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:48.487450 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483710 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:48.487450 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483713 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:48.487450 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483715 4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:48.487450 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483718 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:48.487450 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483721 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:48.487450 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483724 4493 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:48.487450 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.483726 4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:48.487450 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484201 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:48.487450 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484206 4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:48.487951 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484210 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:48.487951 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484214 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:48.487951 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484216 4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:48.487951 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484219 4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:48.487951 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484222 4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:48.487951 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484224 4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:48.487951 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484227 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:48.487951 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484230 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:48.487951 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484232 4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:48.487951 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484235 4493 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:48.487951 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484237 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:48.487951 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484240 4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:48.487951 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484243 4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:48.487951 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484245 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:48.487951 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484248 4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:48.487951 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484251 4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:48.487951 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484254 4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:48.487951 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484256 4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:48.487951 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484259 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:48.488490 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484262 4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:48.488490 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484265 4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:48.488490 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484267 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:48.488490 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484270 4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:48.488490 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484272 4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:48.488490 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484277 4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:48.488490 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484280 4493 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:48.488490 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484283 4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:48.488490 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484286 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:48.488490 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484288 4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:48.488490 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484291 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:48.488490 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484294 4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:48.488490 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484296 4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:48.488490 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484299 4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:48.488490 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484302 4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:48.488490 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484305 4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:48.488490 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484307 4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:48.488490 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484310 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:48.488490 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484312 4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:48.488949 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484315 4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:48.488949 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484317 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:48.488949 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484319 4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:48.488949 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484322 4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:48.488949 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484325 4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:48.488949 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484342 4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:48.488949 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484345 4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:48.488949 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484350 4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:48.488949 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484354 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:48.488949 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484357 4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:48.488949 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484360 4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:48.488949 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484365 4493 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:48.488949 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484368 4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:48.488949 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484370 4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:48.488949 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484373 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:48.488949 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484375 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:48.488949 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484378 4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:48.488949 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484380 4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:48.488949 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484383 4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:48.489444 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484385 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:48.489444 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484388 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:48.489444 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484391 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:48.489444 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484393 4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:48.489444 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484396 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:48.489444 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484398 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:48.489444 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484401 4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:48.489444 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484403 4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:48.489444 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484406 4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:48.489444 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484409 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:48.489444 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484412 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:48.489444 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484414 4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:48.489444 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484417 4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:48.489444 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484419 4493 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:48.489444 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484422 4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:48.489444 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484424 4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:48.489444 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484427 4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:48.489444 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484429 4493 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:48.489444 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484434 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:48.489444 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484436 4493 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:48.489930 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484439 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:48.489930 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484442 4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:48.489930 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484444 4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:48.489930 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484448 4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:48.489930 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484452 4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:48.489930 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484454 4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:48.489930 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484457 4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:48.489930 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484459 4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:48.489930 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484462 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:48.489930 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484464 4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:48.489930 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484467 4493 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:48.489930 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484470 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:48.489930 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484473 4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:48.489930 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484475 4493 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:48.489930 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484477 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:48.489930 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484480 4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:48.489930 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484483 4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:48.489930 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484486 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:48.489930 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484489 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:48.489930 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484491 4493 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:48.490431 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484494 4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:48.490431 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484496 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:48.490431 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484500 4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:48.490431 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484502 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:48.490431 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484505 4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:48.490431 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484508 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:48.490431 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.484510 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:48.490431 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484603 4493 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 20:28:48.490431 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484615 4493 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 20:28:48.490431 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484621 4493 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 20:28:48.490431 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484627 4493 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 20:28:48.490431 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484632 4493 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 20:28:48.490431 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484635 4493 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 20:28:48.490431 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484639 4493 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 20:28:48.490431 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484643 4493 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 20:28:48.490431 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484646 4493 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 20:28:48.490431 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484650 4493 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 20:28:48.490431 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484653 4493 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 20:28:48.490431 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484657 4493 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 20:28:48.490431 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484660 4493 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 20:28:48.490431 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484663 4493 flags.go:64] FLAG: --cgroup-root="" Nov 03 20:28:48.490431 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484666 4493 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 20:28:48.490971 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484669 4493 flags.go:64] FLAG: --client-ca-file="" Nov 03 20:28:48.490971 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484672 4493 flags.go:64] FLAG: --cloud-provider="external" Nov 03 20:28:48.490971 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484674 4493 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 20:28:48.490971 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484678 4493 flags.go:64] FLAG: --cluster-domain="" Nov 03 20:28:48.490971 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484681 4493 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 20:28:48.490971 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484684 4493 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 20:28:48.490971 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484687 4493 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 20:28:48.490971 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484691 4493 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 20:28:48.490971 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484695 4493 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 20:28:48.490971 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484698 4493 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 20:28:48.490971 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484701 4493 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 20:28:48.490971 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484704 4493 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 20:28:48.490971 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484707 4493 flags.go:64] FLAG: --contention-profiling="false" Nov 03 20:28:48.490971 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484711 4493 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 20:28:48.490971 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484714 4493 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 20:28:48.490971 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484718 4493 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 20:28:48.490971 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484721 4493 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 20:28:48.490971 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484725 4493 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 20:28:48.490971 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484728 4493 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 20:28:48.490971 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484731 4493 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 20:28:48.490971 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484735 4493 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 20:28:48.490971 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484738 4493 flags.go:64] FLAG: --enable-server="true" Nov 03 20:28:48.490971 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484741 4493 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 20:28:48.490971 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484745 4493 flags.go:64] FLAG: --event-burst="100" Nov 03 20:28:48.490971 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484749 4493 flags.go:64] FLAG: --event-qps="50" Nov 03 20:28:48.491719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484751 4493 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 20:28:48.491719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484755 4493 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 20:28:48.491719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484758 4493 flags.go:64] FLAG: --eviction-hard="" Nov 03 20:28:48.491719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484762 4493 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 20:28:48.491719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484765 4493 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 20:28:48.491719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484768 4493 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 20:28:48.491719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484772 4493 flags.go:64] FLAG: --eviction-soft="" Nov 03 20:28:48.491719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484774 4493 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 20:28:48.491719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484778 4493 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 20:28:48.491719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484781 4493 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 20:28:48.491719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484784 4493 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 20:28:48.491719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484787 4493 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 20:28:48.491719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484790 4493 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 20:28:48.491719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484792 4493 flags.go:64] FLAG: --feature-gates="" Nov 03 20:28:48.491719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484797 4493 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 20:28:48.491719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484800 4493 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 20:28:48.491719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484803 4493 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 20:28:48.491719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484806 4493 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 20:28:48.491719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484809 4493 flags.go:64] FLAG: --healthz-port="10248" Nov 03 20:28:48.491719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484812 4493 flags.go:64] FLAG: --help="false" Nov 03 20:28:48.491719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484815 4493 flags.go:64] FLAG: --hostname-override="ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:48.491719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484823 4493 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 20:28:48.491719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484826 4493 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 20:28:48.491719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484829 4493 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 20:28:48.492314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484833 4493 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 20:28:48.492314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484836 4493 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 20:28:48.492314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484839 4493 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 20:28:48.492314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484844 4493 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 20:28:48.492314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484847 4493 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 20:28:48.492314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484850 4493 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 20:28:48.492314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484853 4493 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 20:28:48.492314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484857 4493 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 20:28:48.492314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484860 4493 flags.go:64] FLAG: --kube-reserved="" Nov 03 20:28:48.492314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484863 4493 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 20:28:48.492314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484866 4493 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 20:28:48.492314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484870 4493 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 20:28:48.492314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484872 4493 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 20:28:48.492314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484876 4493 flags.go:64] FLAG: --lock-file="" Nov 03 20:28:48.492314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484879 4493 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 20:28:48.492314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484882 4493 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 20:28:48.492314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484885 4493 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 20:28:48.492314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484890 4493 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 20:28:48.492314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484893 4493 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 20:28:48.492314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484896 4493 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 20:28:48.492314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484899 4493 flags.go:64] FLAG: --logging-format="text" Nov 03 20:28:48.492314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484902 4493 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 20:28:48.492314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484905 4493 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 20:28:48.492314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484908 4493 flags.go:64] FLAG: --manifest-url="" Nov 03 20:28:48.492915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484911 4493 flags.go:64] FLAG: --manifest-url-header="" Nov 03 20:28:48.492915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484916 4493 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 20:28:48.492915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484919 4493 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 20:28:48.492915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484923 4493 flags.go:64] FLAG: --max-pods="110" Nov 03 20:28:48.492915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484926 4493 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 20:28:48.492915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484929 4493 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 20:28:48.492915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484932 4493 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 20:28:48.492915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484935 4493 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 20:28:48.492915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484938 4493 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 20:28:48.492915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484941 4493 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 20:28:48.492915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484944 4493 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 20:28:48.492915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484952 4493 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 20:28:48.492915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484955 4493 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 20:28:48.492915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484958 4493 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 20:28:48.492915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484961 4493 flags.go:64] FLAG: --pod-cidr="" Nov 03 20:28:48.492915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484965 4493 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 20:28:48.492915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484968 4493 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 20:28:48.492915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484971 4493 flags.go:64] FLAG: --pods-per-core="0" Nov 03 20:28:48.492915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484974 4493 flags.go:64] FLAG: --port="10250" Nov 03 20:28:48.492915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484977 4493 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 20:28:48.492915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484980 4493 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-049fc227c706e7617" Nov 03 20:28:48.492915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484983 4493 flags.go:64] FLAG: --qos-reserved="" Nov 03 20:28:48.492915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484986 4493 flags.go:64] FLAG: --read-only-port="10255" Nov 03 20:28:48.492915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484989 4493 flags.go:64] FLAG: --register-node="true" Nov 03 20:28:48.492915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484992 4493 flags.go:64] FLAG: --register-with-taints="" Nov 03 20:28:48.492915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484996 4493 flags.go:64] FLAG: --registry-burst="10" Nov 03 20:28:48.493699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.484999 4493 flags.go:64] FLAG: --registry-qps="5" Nov 03 20:28:48.493699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485001 4493 flags.go:64] FLAG: --reserved-cpus="" Nov 03 20:28:48.493699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485005 4493 flags.go:64] FLAG: --reserved-memory="" Nov 03 20:28:48.493699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485008 4493 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 20:28:48.493699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485011 4493 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 20:28:48.493699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485014 4493 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 20:28:48.493699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485017 4493 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 20:28:48.493699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485020 4493 flags.go:64] FLAG: --runonce="false" Nov 03 20:28:48.493699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485023 4493 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 20:28:48.493699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485026 4493 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 20:28:48.493699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485029 4493 flags.go:64] FLAG: --seccomp-default="false" Nov 03 20:28:48.493699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485032 4493 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 20:28:48.493699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485035 4493 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 20:28:48.493699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485039 4493 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 20:28:48.493699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485042 4493 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 20:28:48.493699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485045 4493 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 20:28:48.493699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485048 4493 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 20:28:48.493699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485052 4493 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 20:28:48.493699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485055 4493 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 20:28:48.493699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485058 4493 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 20:28:48.493699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485062 4493 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 20:28:48.493699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485065 4493 flags.go:64] FLAG: --system-cgroups="" Nov 03 20:28:48.493699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485068 4493 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 20:28:48.493699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485073 4493 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 20:28:48.493699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485076 4493 flags.go:64] FLAG: --tls-cert-file="" Nov 03 20:28:48.494651 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485079 4493 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 20:28:48.494651 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485083 4493 flags.go:64] FLAG: --tls-min-version="" Nov 03 20:28:48.494651 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485086 4493 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 20:28:48.494651 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485089 4493 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 20:28:48.494651 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485092 4493 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 20:28:48.494651 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485095 4493 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 20:28:48.494651 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485098 4493 flags.go:64] FLAG: --v="2" Nov 03 20:28:48.494651 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485102 4493 flags.go:64] FLAG: --version="false" Nov 03 20:28:48.494651 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485107 4493 flags.go:64] FLAG: --vmodule="" Nov 03 20:28:48.494651 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485111 4493 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 20:28:48.494651 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485114 4493 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 20:28:48.494651 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485253 4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:48.494651 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485257 4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:48.494651 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485260 4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:48.494651 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485263 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:48.494651 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485266 4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:48.494651 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485269 4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:48.494651 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485271 4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:48.494651 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485274 4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:48.494651 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485277 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:48.494651 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485279 4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:48.494651 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485282 4493 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:48.495467 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485285 4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:48.495467 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485287 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:48.495467 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485292 4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:48.495467 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485295 4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:48.495467 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485298 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:48.495467 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485302 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:48.495467 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485306 4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:48.495467 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485310 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:48.495467 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485313 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:48.495467 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485316 4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:48.495467 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485318 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:48.495467 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485321 4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:48.495467 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485324 4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:48.495467 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485326 4493 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:48.495467 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485342 4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:48.495467 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485345 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:48.495467 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485348 4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:48.495467 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485351 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:48.495467 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485353 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:48.496228 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485356 4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:48.496228 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485359 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:48.496228 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485361 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:48.496228 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485364 4493 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:48.496228 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485366 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:48.496228 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485369 4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:48.496228 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485372 4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:48.496228 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485374 4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:48.496228 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485377 4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:48.496228 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485380 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:48.496228 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485382 4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:48.496228 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485385 4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:48.496228 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485387 4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:48.496228 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485391 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:48.496228 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485393 4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:48.496228 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485397 4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:48.496228 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485400 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:48.496228 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485403 4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:48.496228 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485406 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:48.496228 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485409 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:48.496928 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485412 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:48.496928 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485415 4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:48.496928 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485417 4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:48.496928 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485420 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:48.496928 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485423 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:48.496928 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485425 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:48.496928 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485428 4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:48.496928 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485431 4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:48.496928 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485433 4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:48.496928 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485436 4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:48.496928 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485438 4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:48.496928 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485441 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:48.496928 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485444 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:48.496928 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485446 4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:48.496928 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485449 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:48.496928 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485452 4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:48.496928 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485455 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:48.496928 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485458 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:48.496928 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485461 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:48.497629 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485463 4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:48.497629 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485466 4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:48.497629 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485469 4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:48.497629 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485471 4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:48.497629 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485474 4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:48.497629 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485477 4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:48.497629 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485479 4493 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:48.497629 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485483 4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:48.497629 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485486 4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:48.497629 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485489 4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:48.497629 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485492 4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:48.497629 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485495 4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:48.497629 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485498 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:48.497629 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485501 4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:48.497629 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485503 4493 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:48.497629 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485506 4493 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:48.497629 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485510 4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:48.497629 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485514 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:48.497629 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485517 4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:48.497629 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485520 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:48.498130 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485523 4493 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:48.498130 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485525 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:48.498130 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485528 4493 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:48.498130 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485531 4493 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:48.498130 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485534 4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:48.498130 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485536 4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:48.498130 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485539 4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:48.498130 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485542 4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:48.498130 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485544 4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:48.498130 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485547 4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:48.498130 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485549 4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:48.498130 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485552 4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:48.498130 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485554 4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:48.498130 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485557 4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:48.498130 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485559 4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:48.498130 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485562 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:48.498130 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485565 4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:48.498821 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.485590 4493 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:28:48.498821 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485624 4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:48.498821 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485628 4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:48.498821 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485632 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:48.498821 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485636 4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:48.498821 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485639 4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:48.498821 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485643 4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:48.498821 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485646 4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:48.498821 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485648 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:48.498821 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485651 4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:48.498821 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485654 4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:48.498821 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485656 4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:48.498821 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485659 4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:48.498821 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485662 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:48.498821 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485666 4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:48.498821 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485671 4493 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:48.499489 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485675 4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:48.499489 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485679 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:48.499489 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485683 4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:48.499489 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485687 4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:48.499489 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485691 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:48.499489 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485695 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:48.499489 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485701 4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:48.499489 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485706 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:48.499489 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485709 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:48.499489 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485712 4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:48.499489 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485714 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:48.499489 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485717 4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:48.499489 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485719 4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:48.499489 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485722 4493 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:48.499489 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485725 4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:48.499489 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485728 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:48.499489 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485731 4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:48.499489 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485733 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:48.499489 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485736 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:48.500007 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485739 4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:48.500007 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485744 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:48.500007 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485747 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:48.500007 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485751 4493 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:48.500007 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485753 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:48.500007 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485756 4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:48.500007 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485759 4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:48.500007 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485761 4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:48.500007 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485764 4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:48.500007 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485766 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:48.500007 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485769 4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:48.500007 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485772 4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:48.500007 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485774 4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:48.500007 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485777 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:48.500007 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485779 4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:48.500007 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485782 4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:48.500007 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485785 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:48.500007 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485788 4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:48.500007 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485792 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:48.500007 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485794 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:48.500512 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485797 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:48.500512 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485800 4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:48.500512 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485802 4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:48.500512 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485805 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:48.500512 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485807 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:48.500512 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485810 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:48.500512 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485812 4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:48.500512 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485815 4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:48.500512 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485818 4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:48.500512 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485820 4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:48.500512 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485823 4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:48.500512 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485825 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:48.500512 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485828 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:48.500512 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485832 4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:48.500512 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485838 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:48.500512 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485843 4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:48.500512 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485846 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:48.500512 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485848 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:48.500512 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485851 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:48.501101 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485853 4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:48.501101 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485856 4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:48.501101 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485858 4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:48.501101 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485861 4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:48.501101 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485864 4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:48.501101 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485867 4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:48.501101 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485869 4493 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:48.501101 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485872 4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:48.501101 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485875 4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:48.501101 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485877 4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:48.501101 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485880 4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:48.501101 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485883 4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:48.501101 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485885 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:48.501101 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485888 4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:48.501101 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485891 4493 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:48.501101 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485893 4493 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:48.501101 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485897 4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:48.501101 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485901 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:48.501101 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485904 4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:48.501101 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485907 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:48.501658 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485910 4493 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:48.501658 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485913 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:48.501658 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485915 4493 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:48.501658 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485922 4493 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:48.501658 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485925 4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:48.501658 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485928 4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:48.501658 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485932 4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:48.501658 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485934 4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:48.501658 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485939 4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:48.501658 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485941 4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:48.501658 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485944 4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:48.501658 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485946 4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:48.501658 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.485949 4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:48.501658 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.489816 4493 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 20:28:48.501658 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.489830 4493 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 20:28:48.501658 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.489848 4493 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 20:28:48.501658 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.489853 4493 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 20:28:48.501658 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489916 4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:48.501658 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489921 4493 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:48.501658 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489924 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:48.502343 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489927 4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:48.502343 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489930 4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:48.502343 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489933 4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:48.502343 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489936 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:48.502343 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489939 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:48.502343 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489942 4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:48.502343 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489945 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:48.502343 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489948 4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:48.502343 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489950 4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:48.502343 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489953 4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:48.502343 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489956 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:48.502343 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489959 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:48.502343 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489962 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:48.502343 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489964 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:48.502343 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489969 4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:48.502343 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489973 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:48.502343 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489977 4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:48.502343 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489980 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:48.502805 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489983 4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:48.502805 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489986 4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:48.502805 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489989 4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:48.502805 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489991 4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:48.502805 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489994 4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:48.502805 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.489997 4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:48.502805 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490000 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:48.502805 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490003 4493 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:48.502805 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490006 4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:48.502805 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490008 4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:48.502805 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490011 4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:48.502805 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490014 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:48.502805 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490016 4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:48.502805 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490019 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:48.502805 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490022 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:48.502805 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490024 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:48.502805 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490027 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:48.502805 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490030 4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:48.502805 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490032 4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:48.502805 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490035 4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:48.503289 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490037 4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:48.503289 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490040 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:48.503289 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490043 4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:48.503289 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490045 4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:48.503289 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490048 4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:48.503289 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490050 4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:48.503289 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490053 4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:48.503289 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490056 4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:48.503289 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490058 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:48.503289 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490061 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:48.503289 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490063 4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:48.503289 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490067 4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:48.503289 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490069 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:48.503289 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490072 4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:48.503289 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490074 4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:48.503289 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490078 4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:48.503289 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490080 4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:48.503289 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490083 4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:48.503289 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490086 4493 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:48.503289 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490088 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:48.503795 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490091 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:48.503795 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490094 4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:48.503795 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490096 4493 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:48.503795 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490099 4493 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:48.503795 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490103 4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:48.503795 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490107 4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:48.503795 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490110 4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:48.503795 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490113 4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:48.503795 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490116 4493 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:48.503795 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490118 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:48.503795 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490121 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:48.503795 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490124 4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:48.503795 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490126 4493 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:48.503795 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490129 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:48.503795 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490131 4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:48.503795 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490134 4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:48.503795 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490136 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:48.503795 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490139 4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:48.503795 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490141 4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:48.503795 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490144 4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:48.504284 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490146 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:48.504284 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490149 4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:48.504284 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490152 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:48.504284 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490154 4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:48.504284 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490157 4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:48.504284 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490160 4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:48.504284 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490163 4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:48.504284 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490165 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:48.504284 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490168 4493 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:48.504284 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490171 4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:48.504284 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490174 4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:48.504284 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490177 4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:48.504284 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490179 4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:48.504284 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490182 4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:48.504284 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490185 4493 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:48.504284 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490187 4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:48.504284 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490190 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:48.504284 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490192 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:48.504284 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490195 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:48.504284 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490197 4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:48.504838 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490200 4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:48.504838 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490202 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:48.504838 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490205 4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:48.504838 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490207 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:48.504838 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490210 4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:48.504838 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.490239 4493 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:28:48.504838 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490394 4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:48.504838 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490401 4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:48.504838 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490404 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:48.504838 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490407 4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:48.504838 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490410 4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:48.504838 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490412 4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:48.504838 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490415 4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:48.504838 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490418 4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:48.504838 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490420 4493 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:48.504838 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490423 4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:48.505224 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490426 4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:48.505224 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490428 4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:48.505224 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490431 4493 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:48.505224 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490433 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:48.505224 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490436 4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:48.505224 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490439 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:48.505224 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490443 4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:48.505224 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490445 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:48.505224 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490448 4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:48.505224 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490450 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:48.505224 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490453 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:48.505224 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490456 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:48.505224 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490458 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:48.505224 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490461 4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:48.505224 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490463 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:48.505224 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490466 4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:48.505224 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490468 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:48.505224 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490471 4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:48.505224 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490474 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:48.505700 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490476 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:48.505700 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490480 4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:48.505700 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490484 4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:48.505700 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490487 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:48.505700 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490490 4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:48.505700 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490493 4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:48.505700 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490497 4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:48.505700 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490499 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:48.505700 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490502 4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:48.505700 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490504 4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:48.505700 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490507 4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:48.505700 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490509 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:48.505700 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490512 4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:48.505700 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490515 4493 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:48.505700 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490518 4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:48.505700 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490520 4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:48.505700 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490523 4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:48.505700 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490526 4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:48.505700 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490528 4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:48.506194 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490531 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:48.506194 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490534 4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:48.506194 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490536 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:48.506194 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490539 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:48.506194 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490541 4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:48.506194 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490544 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:48.506194 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490546 4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:48.506194 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490549 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:48.506194 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490551 4493 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:48.506194 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490554 4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:48.506194 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490556 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:48.506194 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490559 4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:48.506194 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490561 4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:48.506194 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490564 4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:48.506194 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490566 4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:48.506194 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490573 4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:48.506194 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490576 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:48.506194 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490578 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:48.506194 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490581 4493 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:48.506194 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490583 4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:48.506697 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490586 4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:48.506697 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490589 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:48.506697 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490591 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:48.506697 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490594 4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:48.506697 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490597 4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:48.506697 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490599 4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:48.506697 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490602 4493 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:48.506697 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490604 4493 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:48.506697 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490607 4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:48.506697 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490611 4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:48.506697 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490614 4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:48.506697 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490617 4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:48.506697 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490620 4493 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:48.506697 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490623 4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:48.506697 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490626 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:48.506697 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490629 4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:48.506697 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490631 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:48.506697 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490634 4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:48.506697 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490637 4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:48.506697 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490640 4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:48.507157 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490642 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:48.507157 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490645 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:48.507157 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490647 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:48.507157 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490650 4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:48.507157 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490653 4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:48.507157 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490655 4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:48.507157 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490658 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:48.507157 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490662 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:48.507157 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490665 4493 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:48.507157 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490667 4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:48.507157 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490670 4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:48.507157 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490673 4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:48.507157 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490675 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:48.507157 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490678 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:48.507157 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490681 4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:48.507157 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490683 4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:48.507157 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490686 4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:48.507157 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490689 4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:48.507724 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.490712 4493 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:28:48.507724 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490744 4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:48.507724 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490748 4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:48.507724 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490751 4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:48.507724 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490754 4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:48.507724 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490757 4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:48.507724 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490760 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:48.507724 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490762 4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:48.507724 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490765 4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:48.507724 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490768 4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:48.507724 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490770 4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:48.507724 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490773 4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:48.507724 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490776 4493 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:48.507724 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490778 4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:48.507724 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490780 4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:48.507724 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490783 4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:48.508150 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490785 4493 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:48.508150 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490788 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:48.508150 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490791 4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:48.508150 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490793 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:48.508150 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490796 4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:48.508150 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490798 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:48.508150 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490802 4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:48.508150 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490805 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:48.508150 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490808 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:48.508150 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490810 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:48.508150 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490813 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:48.508150 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490816 4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:48.508150 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490818 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:48.508150 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490821 4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:48.508150 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490824 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:48.508150 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490827 4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:48.508150 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490830 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:48.508150 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490832 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:48.508931 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490836 4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:48.508931 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490840 4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:48.508931 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490843 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:48.508931 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490845 4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:48.508931 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490848 4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:48.508931 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490851 4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:48.508931 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490853 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:48.508931 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490856 4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:48.508931 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490859 4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:48.508931 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490861 4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:48.508931 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490864 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:48.508931 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490867 4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:48.508931 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490869 4493 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:48.508931 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490872 4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:48.508931 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490874 4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:48.508931 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490877 4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:48.508931 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490880 4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:48.508931 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490882 4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:48.508931 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490884 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:48.509803 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490887 4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:48.509803 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490891 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:48.509803 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490894 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:48.509803 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490896 4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:48.509803 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490899 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:48.509803 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490901 4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:48.509803 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490904 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:48.509803 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490907 4493 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:48.509803 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490910 4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:48.509803 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490912 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:48.509803 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490915 4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:48.509803 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490917 4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:48.509803 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490920 4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:48.509803 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490922 4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:48.509803 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490925 4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:48.509803 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490927 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:48.509803 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490930 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:48.509803 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490933 4493 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:48.509803 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490935 4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:48.509803 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490938 4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:48.510677 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490940 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:48.510677 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490943 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:48.510677 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490945 4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:48.510677 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490948 4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:48.510677 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490951 4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:48.510677 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490953 4493 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:48.510677 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490956 4493 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:48.510677 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490959 4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:48.510677 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490963 4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:48.510677 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490966 4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:48.510677 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490969 4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:48.510677 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490972 4493 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:48.510677 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490975 4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:48.510677 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490979 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:48.510677 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490982 4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:48.510677 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490985 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:48.510677 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490988 4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:48.510677 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490990 4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:48.510677 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490993 4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:48.510677 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490996 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:48.511567 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.490999 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:48.511567 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.491001 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:48.511567 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.491004 4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:48.511567 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.491007 4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:48.511567 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.491009 4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:48.511567 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.491012 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:48.511567 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.491014 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:48.511567 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.491017 4493 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:48.511567 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.491020 4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:48.511567 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.491024 4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:48.511567 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.491026 4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:48.511567 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.491029 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:48.511567 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.491032 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:48.511567 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:28:48.491035 4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:48.511567 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.491083 4493 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 20:28:48.511567 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.492051 4493 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 20:28:48.511567 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.492105 4493 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 20:28:48.511567 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.492285 4493 server.go:1014] "Starting client certificate rotation" Nov 03 20:28:48.511567 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.492427 4493 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:28:48.512381 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.492486 4493 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 20:23:08 +0000 UTC" deadline="2028-05-05 15:09:18.82678511 +0000 UTC" Nov 03 20:28:48.512381 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.492513 4493 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13170h40m30.33427453s" Nov 03 20:28:48.512381 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.492720 4493 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:28:48.512381 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.493763 4493 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:28:48.512381 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.496825 4493 log.go:25] "Validated CRI v1 runtime API" Nov 03 20:28:48.512381 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.498469 4493 log.go:25] "Validated CRI v1 image API" Nov 03 20:28:48.512381 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.499225 4493 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 20:28:48.512381 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.500756 4493 fs.go:135] Filesystem UUIDs: map[2b4c1d62-c9a2-48f5-91ed-b06e6eb81532:/dev/nvme0n1p4 6ff11b39-2cad-4870-9d97-2394c1f6a523:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 03 20:28:48.512746 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.500772 4493 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c5b6a1b1f30d3e9122ef65b5a92319eaecd4674ea7acfb1194edbbb5ea43818/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c5b6a1b1f30d3e9122ef65b5a92319eaecd4674ea7acfb1194edbbb5ea43818/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/14f0e7044649ef39b41f8df4f11301f5b107fb03376661ebd8b18605a9953596/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/14f0e7044649ef39b41f8df4f11301f5b107fb03376661ebd8b18605a9953596/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/195d211fa9105149de97f3e991e074de3fbefa75aa528ef0e4604e3b4848fca3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/195d211fa9105149de97f3e991e074de3fbefa75aa528ef0e4604e3b4848fca3/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/444bb3bce81beb9241034c1e8c8a8e76d27752a99c32addfbcc916c53563a2e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/444bb3bce81beb9241034c1e8c8a8e76d27752a99c32addfbcc916c53563a2e1/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/752a7a52453719ea41ed2a9926482b8016ac42c1c089927f0d5a4a92fec9d9f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/752a7a52453719ea41ed2a9926482b8016ac42c1c089927f0d5a4a92fec9d9f2/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/764a370bdff5d09b8d06ef47240c6223aae800dac58d02e400dd722008e4da8e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/764a370bdff5d09b8d06ef47240c6223aae800dac58d02e400dd722008e4da8e/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8426b10369feb5597e03c517e9e3d9a6a745ea14724782cf2b4b6bb792b1c120/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8426b10369feb5597e03c517e9e3d9a6a745ea14724782cf2b4b6bb792b1c120/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/916ecd53f7745d6f5bc1f3c7ab629fc90d57467bc749b5a76379d7ecedfcadc1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/916ecd53f7745d6f5bc1f3c7ab629fc90d57467bc749b5a76379d7ecedfcadc1/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/92af30685c6d43145a57276b4e57ebc43fb507c107b145fddbf83cd0120d7867/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/92af30685c6d43145a57276b4e57ebc43fb507c107b145fddbf83cd0120d7867/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a852d31c4b868d05ab7e57a0691f02e1a5c4db0da405f0896605437528d9955c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a852d31c4b868d05ab7e57a0691f02e1a5c4db0da405f0896605437528d9955c/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0742edfa33aa6021b9dc3a08e45c023e436ce141fd7b15572b3831f1a5a7c66/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0742edfa33aa6021b9dc3a08e45c023e436ce141fd7b15572b3831f1a5a7c66/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7b7f04d8d2fa36cc327714791ba9c625aa82b9acd636b3978179d64ddc92f70/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7b7f04d8d2fa36cc327714791ba9c625aa82b9acd636b3978179d64ddc92f70/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7ec26740a2688262054a179f78c6e66b34689da2e54e2cabb3df4d1b835f026/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7ec26740a2688262054a179f78c6e66b34689da2e54e2cabb3df4d1b835f026/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0be4d66f0bdf0aac69f183f27d9edb3ba94d26257901efec80d7d7c8eba467e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0be4d66f0bdf0aac69f183f27d9edb3ba94d26257901efec80d7d7c8eba467e/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f62cdd8d3dd5a0439afe7c3e7cb27b1c794287bb49d5dd8f9deb51158f2a315b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f62cdd8d3dd5a0439afe7c3e7cb27b1c794287bb49d5dd8f9deb51158f2a315b/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2281c3da-747c-4a37-89ec-b764d8a95405/volumes/kubernetes.io~projected/kube-api-access-9487d:{mountpoint:/var/lib/kubelet/pods/2281c3da-747c-4a37-89ec-b764d8a95405/volumes/kubernetes.io~projected/kube-api-access-9487d major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34b6b405-a677-4fef-8ee6-37d84a1c5902/volumes/kubernetes.io~projected/kube-api-access-h9qrh:{mountpoint:/var/lib/kubelet/pods/34b6b405-a677-4fef-8ee6-37d84a1c5902/volumes/kubernetes.io~projected/kube-api-access-h9qrh major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3514724f-deea-4b4e-bce9-48ce7862d6f0/volumes/kubernetes.io~projected/kube-api-access-hwv4c:{mountpoint:/var/lib/kubelet/pods/3514724f-deea-4b4e-bce9-48ce7862d6f0/volumes/kubernetes.io~projected/kube-api-access-hwv4c major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/480d6b9d-4f58-4779-a2cb-aa4b64f93881/volumes/kubernetes.io~projected/kube-api-access-2m49g:{mountpoint:/var/lib/kubelet/pods/480d6b9d-4f58-4779-a2cb-aa4b64f93881/volumes/kubernetes.io~projected/kube-api-access-2m49g major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4cc66ac6-6311-4144-8db1-3c39182e1a19/volumes/kubernetes.io~projected/kube-api-access-vdhpf:{mountpoint:/var/lib/kubelet/pods/4cc66ac6-6311-4144-8db1-3c39182e1a19/volumes/kubernetes.io~projected/kube-api-access-vdhpf major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59643abf-923a-4d4d-8235-c2ad22c8aea0/volumes/kubernetes.io~projected/kube-api-access-wfrc5:{mountpoint:/var/lib/kubelet/pods/59643abf-923a-4d4d-8235-c2ad22c8aea0/volumes/kubernetes.io~projected/kube-api-access-wfrc5 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6cd2ab07-4196-4ed1-b357-38f01b99b71d/volumes/kubernetes.io~projected/kube-api-access-6nlpn:{mountpoint:/var/lib/kubelet/pods/6cd2ab07-4196-4ed1-b357-38f01b99b71d/volumes/kubernetes.io~projected/kube-api-access-6nlpn major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e65f973-499a-4dac-af00-b485fb5dbbfe/volumes/kubernetes.io~projected/kube-api-access-c6ls8:{mountpoint:/var/lib/kubelet/pods/7e65f973-499a-4dac-af00-b485fb5dbbfe/volumes/kubernetes.io~projected/kube-api-access-c6ls8 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e65f973-499a-4dac-af00-b485fb5dbbfe/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/7e65f973-499a-4dac-af00-b485fb5dbbfe/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9844c086-efc5-46f3-9d37-34c2fa013f34/volumes/kubernetes.io~projected/kube-api-access-zrs5m:{mountpoint:/var/lib/kubelet/pods/9844c086-efc5-46f3-9d37-34c2fa013f34/volumes/kubernetes.io~projected/kube-api-access-zrs5m major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3868c3d-8d3a-4c40-8821-ca116f8dcfed/volumes/kubernetes.io~projected/kube-api-access-4fhqp:{mountpoint:/var/lib/kubelet/pods/a3868c3d-8d3a-4c40-8821-ca116f8dcfed/volumes/kubernetes.io~projected/kube-api-access-4fhqp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b013faae-082b-495b-9d77-b4371e82e379/volumes/kubernetes.io~projected/kube-api-access-szxvh:{mountpoint:/var/lib/kubelet/pods/b013faae-082b-495b-9d77-b4371e82e379/volumes/kubernetes.io~projected/kube-api-access-szxvh major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5474343-621e-4dd9-948c-ad71bd6f00c2/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/c5474343-621e-4dd9-948c-ad71bd6f00c2/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5474343-621e-4dd9-948c-ad71bd6f00c2/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/c5474343-621e-4dd9-948c-ad71bd6f00c2/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5474343-621e-4dd9-948c-ad71bd6f00c2/volumes/kubernetes.io~projected/kube-api-access-s62xs:{mountpoint:/var/lib/kubelet/pods/c5474343-621e-4dd9-948c-ad71bd6f00c2/volumes/kubernetes.io~projected/kube-api-access-s62xs major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d48c0b65-db57-43b2-8e7c-6e9e8feddf89/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d48c0b65-db57-43b2-8e7c-6e9e8feddf89/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d48c0b65-db57-43b2-8e7c-6e9e8feddf89/volumes/kubernetes.io~projected/kube-api-access-bmnw9:{mountpoint:/var/lib/kubelet/pods/d48c0b65-db57-43b2-8e7c-6e9e8feddf89/volumes/kubernetes.io~projected/kube-api-access-bmnw9 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d48c0b65-db57-43b2-8e7c-6e9e8feddf89/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d48c0b65-db57-43b2-8e7c-6e9e8feddf89/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5125fef-6a2d-45d6-99d2-90915b575f32/volumes/kubernetes.io~projected/kube-api-access-hrlqh:{mountpoint:/var/lib/kubelet/pods/d5125fef-6a2d-45d6-99d2-90915b575f32/volumes/kubernetes.io~projected/kube-api-access-hrlqh major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e99c666f-7bb3-4856-a2fc-935ee184dae3/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/e99c666f-7bb3-4856-a2fc-935ee184dae3/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef30d813-b7ed-417d-a9fe-85118bf77a95/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ef30d813-b7ed-417d-a9fe-85118bf77a95/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7ab819a-4321-4958-a5d5-230cbdbd9428/volumes/kubernetes.io~projected/kube-api-access-8ss5t:{mountpoint:/var/lib/kubelet/pods/f7ab819a-4321-4958-a5d5-230cbdbd9428/volumes/kubernetes.io~projected/kube-api-access-8ss5t major:0 minor:63 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/47e766389a274b2dacc5f7a608a0f70b67596f9bb7c0efe07390cffffbe5ae95/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/adfbb7f6fb6ad6efcbd49424eb28fdf8c53be5b5bc88ef5e26a0d725a1a12098/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/fc55458c0f4445454a87da5d9b055177f49243582007424df6e8947642d2f436/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/124108ee83ab56c9b57530adf6f1582b7711a030f7e1a837aac02b537565d731/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-113:{mountpoint:/var/lib/containers/storage/overlay/e914fe78b0a4b5909cc0cfebb82e9bf47c79c7c839596fb4a5337bb7471f3de6/merged major:0 minor:113 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/5c7f9fb135eaf3535ff11c57918e0c4ea45e8d93ddf2749ff450ca42beb2df55/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/c1c4c0d66d768078f0ebb87800ad92fc70f48f67880bb4509187ad982deabd9c/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/8677852a232ee53402dd6ab9a207a2a9f1da348cfc333490326d11c0e2c43a4e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/3b35d3a9c64e2fc10971c8116b41d681650a47a94ddf951ee3796257fdc33970/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/0363ff88a5bf441ee5328a326af9290cfb672e3cfa2a4b0c83d5645ac7935c3f/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/0e9392e4e6edffb0c13a7ae2a8fdc263d6721ee0cbeb271f563d132c94047959/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/971ec9d30145122b8add61ca661312ee202b8127f9a7c894a666d5cbe8f1d0f4/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/80ee1e529218e01003cf3bd2e24f4519e3b33d6ce0a487443f691fc3fe5b680a/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/157cb365d793db79e9165171ff47e999c234fbf5fafe395735c2d8d4558085db/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/52df0ac662b4ec5bef84aa0151599c032cbc9172a1625de3bfabcc1fcde681a6/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/af56e85f0eaaef15f529fde056d758bc938a38829d109204f6fa566a291d3f3f/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/fd85cc048cae301b6d0219ecde41412b22f33a7dcfb8d408815eed3bb8f04b4d/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/9fc656783c0c0ffc15d0de662919d94a23b6fa27130391e1565ef6e0da7fa004/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/ce9c1c0f8944ca0c332085439e7f3e060d4daa6ec4efd7ba6bc7adbd9ed54e32/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/ff145ae9fe30ff65fe4ad6f76a039b2de93ff426bebe8d1e6a98205e53f750bd/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/b72934bf82651ba4b882e1796cc44f2dfe4330e9ac520127646707ab52a1c5a7/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/dc060d9000c7820ab822af10cf945e0b1bdf539c706eadea29652d6f95209309/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/ba0d382b6376c46970876ba28d848582db4cda0fc0544c371fd812c923e62f88/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/b72f3386bc2b95abb3c05c90742fd671dc4dbfab5e55477e73004abfb8b11eba/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/5c6315e4556cdd30707abd7b03f3bf5562ef589922a17b92be2d4e10012aefe1/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/4ea82cc960ba8cda3ddd2bd4c29a2417aa8b7d571598d1db63feea1c29f89c2c/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/524144aa19a2df481ec7c2abe67828618a5f68327d901e69b17f57e189fc3eb7/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/4d31a2277e719d9cc0f7ea3de733c817265a276f39824127eb71d7c8c22a7628/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/0ba12f9e311d058c894eac3f257e9e3e42f8af9071a574756c045848210453d5/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/6c83e10208523650a2b410b88a5b585cfd54e26d06ef34efdf96275a69bba594/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/7e5476ca9d966643baa1475b7c99a51c432094b9845595e496b7f5d853f33572/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/0e2aa2b53136ae49e5c24d18a77f5b381ab3f92d967d1a7a98f43919e5bbc2b0/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/397be107ad0091b94d7c0111dd36a71ba6793d7fd6b50bec54cdd8539c74ffb6/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/b1695b0fb547fa992802dcf9cb356d3273b3c2157d58c43cf8e87c98abead7aa/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/b6cf2d326547a35b2950904d422f64ec8931fe26b76ab5908934678eddf68d17/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/e24c46cdb9a6c6b107dea493bfde2d7d47754e343bc981c5ae8d59f80760183a/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/aed890f406ab586723235d3f20393e4d13b9d66d4c26aec4906503e36fd6308a/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/60b8588d99ee83d277856371192c182c09814f0f947626eab800491fbb1abb06/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/93d29e8596d7fe93a85dfd01e8374baea7ee3a5813f56af28da9f0c7c7c25236/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/029aabeaa5bb73c631cf9d466ef4767437e41a8b6dd0ba392dedf2e7c1064bf6/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/72fb4a5b5c465c3e442ad0cd236b15acce0e4f2bafb8533ca6a53b63c108af86/merged major:0 minor:97 fsType:overlay blockSize:0}] Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.507104 4493 manager.go:217] Machine: {Timestamp:2026-11-03 20:28:48.506663233 +0000 UTC m=+0.064653724 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100235 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26fb0d2e4b72d42008dd6094eb0d31 SystemUUID:ec26fb0d-2e4b-72d4-2008-dd6094eb0d31 BootID:18484d08-debf-4fda-8c05-b01468453a28 Filesystems:[{Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d48c0b65-db57-43b2-8e7c-6e9e8feddf89/volumes/kubernetes.io~projected/kube-api-access-bmnw9 DeviceMajor:0 DeviceMinor:65 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a852d31c4b868d05ab7e57a0691f02e1a5c4db0da405f0896605437528d9955c/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c5474343-621e-4dd9-948c-ad71bd6f00c2/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9844c086-efc5-46f3-9d37-34c2fa013f34/volumes/kubernetes.io~projected/kube-api-access-zrs5m DeviceMajor:0 DeviceMinor:60 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/14f0e7044649ef39b41f8df4f11301f5b107fb03376661ebd8b18605a9953596/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e65f973-499a-4dac-af00-b485fb5dbbfe/volumes/kubernetes.io~projected/kube-api-access-c6ls8 DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b013faae-082b-495b-9d77-b4371e82e379/volumes/kubernetes.io~projected/kube-api-access-szxvh DeviceMajor:0 DeviceMinor:216 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c5474343-621e-4dd9-948c-ad71bd6f00c2/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0742edfa33aa6021b9dc3a08e45c023e436ce141fd7b15572b3831f1a5a7c66/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/59643abf-923a-4d4d-8235-c2ad22c8aea0/volumes/kubernetes.io~projected/kube-api-access-wfrc5 DeviceMajor:0 DeviceMinor:62 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7b7f04d8d2fa36cc327714791ba9c625aa82b9acd636b3978179d64ddc92f70/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/480d6b9d-4f58-4779-a2cb-aa4b64f93881/volumes/kubernetes.io~projected/kube-api-access-2m49g DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/f7ab819a-4321-4958-a5d5-230cbdbd9428/volumes/kubernetes.io~projected/kube-api-access-8ss5t DeviceMajor:0 DeviceMinor:63 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7ec26740a2688262054a179f78c6e66b34689da2e54e2cabb3df4d1b835f026/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef30d813-b7ed-417d-a9fe-85118bf77a95/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0be4d66f0bdf0aac69f183f27d9edb3ba94d26257901efec80d7d7c8eba467e/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/752a7a52453719ea41ed2a9926482b8016ac42c1c089927f0d5a4a92fec9d9f2/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/916ecd53f7745d6f5bc1f3c7ab629fc90d57467bc749b5a76379d7ecedfcadc1/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f62cdd8d3dd5a0439afe7c3e7cb27b1c794287bb49d5dd8f9deb51158f2a315b/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/34b6b405-a677-4fef-8ee6-37d84a1c5902/volumes/kubernetes.io~projected/kube-api-access-h9qrh DeviceMajor:0 DeviceMinor:219 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d5125fef-6a2d-45d6-99d2-90915b575f32/volumes/kubernetes.io~projected/kube-api-access-hrlqh DeviceMajor:0 DeviceMinor:64 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/c5474343-621e-4dd9-948c-ad71bd6f00c2/volumes/kubernetes.io~projected/kube-api-access-s62xs DeviceMajor:0 DeviceMinor:66 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/195d211fa9105149de97f3e991e074de3fbefa75aa528ef0e4604e3b4848fca3/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d48c0b65-db57-43b2-8e7c-6e9e8feddf89/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8426b10369feb5597e03c517e9e3d9a6a745ea14724782cf2b4b6bb792b1c120/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c5b6a1b1f30d3e9122ef65b5a92319eaecd4674ea7acfb1194edbbb5ea43818/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/92af30685c6d43145a57276b4e57ebc43fb507c107b145fddbf83cd0120d7867/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e65f973-499a-4dac-af00-b485fb5dbbfe/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/444bb3bce81beb9241034c1e8c8a8e76d27752a99c32addfbcc916c53563a2e1/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3514724f-deea-4b4e-bce9-48ce7862d6f0/volumes/kubernetes.io~projected/kube-api-access-hwv4c DeviceMajor:0 DeviceMinor:228 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d48c0b65-db57-43b2-8e7c-6e9e8feddf89/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/764a370bdff5d09b8d06ef47240c6223aae800dac58d02e400dd722008e4da8e/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/a3868c3d-8d3a-4c40-8821-ca116f8dcfed/volumes/kubernetes.io~projected/kube-api-access-4fhqp DeviceMajor:0 DeviceMinor:61 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e99c666f-7bb3-4856-a2fc-935ee184dae3/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4cc66ac6-6311-4144-8db1-3c39182e1a19/volumes/kubernetes.io~projected/kube-api-access-vdhpf DeviceMajor:0 DeviceMinor:67 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-113 DeviceMajor:0 DeviceMinor:113 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6cd2ab07-4196-4ed1-b357-38f01b99b71d/volumes/kubernetes.io~projected/kube-api-access-6nlpn DeviceMajor:0 DeviceMinor:207 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2281c3da-747c-4a37-89ec-b764d8a95405/volumes/kubernetes.io~projected/kube-api-access-9487d DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:14f0e7044649ef3 MacAddress:46:16:9d:f7:41:95 Speed:10000 Mtu:8901} {Name:444bb3bce81beb9 MacAddress:32:01:30:36:fa:27 Speed:10000 Mtu:8901} {Name:8426b10369feb55 MacAddress:ba:eb:77:44:31:b5 Speed:10000 Mtu:8901} {Name:916ecd53f7745d6 MacAddress:c2:53:03:8d:97:80 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:c1:3c:25:81 Speed:0 Mtu:9001} {Name:br-int MacAddress:9a:5f:1a:68:ec:83 Speed:0 Mtu:8901} {Name:ens5 MacAddress:06:ff:c1:3c:25:81 Speed:0 Mtu:9001} {Name:f62cdd8d3dd5a04 MacAddress:9a:d4:0c:b8:33:cb Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:ae:0f:fe:a2:fe:e8 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:7a:df:55:fb:e3:e1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.507369 4493 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.507412 4493 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.507654 4493 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.507673 4493 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-26.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.507812 4493 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.507819 4493 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.507832 4493 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.507846 4493 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.507918 4493 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.508064 4493 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.508167 4493 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.508199 4493 kubelet.go:493] "Attempting to sync node with API server" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.508213 4493 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.508222 4493 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.508231 4493 kubelet.go:405] "Adding apiserver pod source" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.508242 4493 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.508897 4493 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.508907 4493 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.509379 4493 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.511370 4493 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.511379 4493 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.511466 4493 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.511684 4493 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.511712 4493 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.511806 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.511821 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.511831 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.511843 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.511856 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.511867 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.511877 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.511884 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.511898 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.511913 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.511926 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.511939 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.511978 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 20:28:48.512976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.511988 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 20:28:48.515589 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.515574 4493 server.go:1263] "Started kubelet" Nov 03 20:28:48.515742 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.515683 4493 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 20:28:48.516670 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.516064 4493 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 20:28:48.516670 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.516178 4493 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 20:28:48.516549 ip-10-0-138-26 systemd[1]: Started Kubernetes Kubelet. Nov 03 20:28:48.517990 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.517974 4493 server.go:317] "Adding debug handlers to kubelet server" Nov 03 20:28:48.524064 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.524044 4493 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 20:28:48.525323 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.525303 4493 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 20:28:48.525455 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.525435 4493 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 20:28:48.525455 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.525450 4493 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 20:28:48.525455 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.525446 4493 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 20:28:48.525666 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.525499 4493 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:23:10 +0000 UTC" deadline="2028-05-03 16:08:08.942316143 +0000 UTC" Nov 03 20:28:48.525666 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.525516 4493 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13123h39m20.416802555s" Nov 03 20:28:48.525666 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.525551 4493 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 20:28:48.525666 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:48.525549 4493 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-26.us-west-2.compute.internal\" not found" Nov 03 20:28:48.527267 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527231 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34b6b405-a677-4fef-8ee6-37d84a1c5902" volumeName="kubernetes.io/configmap/34b6b405-a677-4fef-8ee6-37d84a1c5902-service-ca-bundle" seLinuxMountContext="" Nov 03 20:28:48.527376 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527271 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9844c086-efc5-46f3-9d37-34c2fa013f34" volumeName="kubernetes.io/configmap/9844c086-efc5-46f3-9d37-34c2fa013f34-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 20:28:48.527376 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527287 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5474343-621e-4dd9-948c-ad71bd6f00c2" volumeName="kubernetes.io/empty-dir/c5474343-621e-4dd9-948c-ad71bd6f00c2-tmp" seLinuxMountContext="" Nov 03 20:28:48.527376 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527299 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34b6b405-a677-4fef-8ee6-37d84a1c5902" volumeName="kubernetes.io/projected/34b6b405-a677-4fef-8ee6-37d84a1c5902-kube-api-access-h9qrh" seLinuxMountContext="" Nov 03 20:28:48.527376 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527312 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="480d6b9d-4f58-4779-a2cb-aa4b64f93881" volumeName="kubernetes.io/projected/480d6b9d-4f58-4779-a2cb-aa4b64f93881-kube-api-access-2m49g" seLinuxMountContext="" Nov 03 20:28:48.527376 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527324 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef30d813-b7ed-417d-a9fe-85118bf77a95" volumeName="kubernetes.io/secret/ef30d813-b7ed-417d-a9fe-85118bf77a95-original-pull-secret" seLinuxMountContext="" Nov 03 20:28:48.527376 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527360 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cc66ac6-6311-4144-8db1-3c39182e1a19" volumeName="kubernetes.io/configmap/4cc66ac6-6311-4144-8db1-3c39182e1a19-iptables-alerter-script" seLinuxMountContext="" Nov 03 20:28:48.527376 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527372 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59643abf-923a-4d4d-8235-c2ad22c8aea0" volumeName="kubernetes.io/empty-dir/59643abf-923a-4d4d-8235-c2ad22c8aea0-tmp-dir" seLinuxMountContext="" Nov 03 20:28:48.527667 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527385 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9844c086-efc5-46f3-9d37-34c2fa013f34" volumeName="kubernetes.io/configmap/9844c086-efc5-46f3-9d37-34c2fa013f34-cni-binary-copy" seLinuxMountContext="" Nov 03 20:28:48.527667 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527397 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5125fef-6a2d-45d6-99d2-90915b575f32" volumeName="kubernetes.io/projected/d5125fef-6a2d-45d6-99d2-90915b575f32-kube-api-access-hrlqh" seLinuxMountContext="" Nov 03 20:28:48.527667 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527411 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34b6b405-a677-4fef-8ee6-37d84a1c5902" volumeName="kubernetes.io/configmap/34b6b405-a677-4fef-8ee6-37d84a1c5902-trusted-ca-bundle" seLinuxMountContext="" Nov 03 20:28:48.527667 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527436 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34b6b405-a677-4fef-8ee6-37d84a1c5902" volumeName="kubernetes.io/empty-dir/34b6b405-a677-4fef-8ee6-37d84a1c5902-archives-path" seLinuxMountContext="" Nov 03 20:28:48.527667 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527449 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="480d6b9d-4f58-4779-a2cb-aa4b64f93881" volumeName="kubernetes.io/configmap/480d6b9d-4f58-4779-a2cb-aa4b64f93881-config-volume" seLinuxMountContext="" Nov 03 20:28:48.527667 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527492 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cd2ab07-4196-4ed1-b357-38f01b99b71d" volumeName="kubernetes.io/projected/6cd2ab07-4196-4ed1-b357-38f01b99b71d-kube-api-access-6nlpn" seLinuxMountContext="" Nov 03 20:28:48.527667 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527509 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9844c086-efc5-46f3-9d37-34c2fa013f34" volumeName="kubernetes.io/configmap/9844c086-efc5-46f3-9d37-34c2fa013f34-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 20:28:48.527667 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527521 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d48c0b65-db57-43b2-8e7c-6e9e8feddf89" volumeName="kubernetes.io/configmap/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-ovnkube-config" seLinuxMountContext="" Nov 03 20:28:48.527667 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527630 4493 factory.go:153] Registering CRI-O factory Nov 03 20:28:48.527667 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527649 4493 factory.go:223] Registration of the crio container factory successfully Nov 03 20:28:48.527969 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527679 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e99c666f-7bb3-4856-a2fc-935ee184dae3" volumeName="kubernetes.io/secret/e99c666f-7bb3-4856-a2fc-935ee184dae3-agent-certs" seLinuxMountContext="" Nov 03 20:28:48.527969 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527703 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b013faae-082b-495b-9d77-b4371e82e379" volumeName="kubernetes.io/empty-dir/b013faae-082b-495b-9d77-b4371e82e379-data-volume" seLinuxMountContext="" Nov 03 20:28:48.527969 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527716 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5125fef-6a2d-45d6-99d2-90915b575f32" volumeName="kubernetes.io/configmap/d5125fef-6a2d-45d6-99d2-90915b575f32-multus-daemon-config" seLinuxMountContext="" Nov 03 20:28:48.527969 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527729 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2281c3da-747c-4a37-89ec-b764d8a95405" volumeName="kubernetes.io/projected/2281c3da-747c-4a37-89ec-b764d8a95405-kube-api-access-9487d" seLinuxMountContext="" Nov 03 20:28:48.527969 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527743 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="480d6b9d-4f58-4779-a2cb-aa4b64f93881" volumeName="kubernetes.io/empty-dir/480d6b9d-4f58-4779-a2cb-aa4b64f93881-tmp-dir" seLinuxMountContext="" Nov 03 20:28:48.527969 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527755 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cc66ac6-6311-4144-8db1-3c39182e1a19" volumeName="kubernetes.io/projected/4cc66ac6-6311-4144-8db1-3c39182e1a19-kube-api-access-vdhpf" seLinuxMountContext="" Nov 03 20:28:48.527969 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527770 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59643abf-923a-4d4d-8235-c2ad22c8aea0" volumeName="kubernetes.io/projected/59643abf-923a-4d4d-8235-c2ad22c8aea0-kube-api-access-wfrc5" seLinuxMountContext="" Nov 03 20:28:48.527969 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527805 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e65f973-499a-4dac-af00-b485fb5dbbfe" volumeName="kubernetes.io/secret/7e65f973-499a-4dac-af00-b485fb5dbbfe-cert" seLinuxMountContext="" Nov 03 20:28:48.527969 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527816 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9844c086-efc5-46f3-9d37-34c2fa013f34" volumeName="kubernetes.io/projected/9844c086-efc5-46f3-9d37-34c2fa013f34-kube-api-access-zrs5m" seLinuxMountContext="" Nov 03 20:28:48.527969 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527829 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3868c3d-8d3a-4c40-8821-ca116f8dcfed" volumeName="kubernetes.io/projected/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-kube-api-access-4fhqp" seLinuxMountContext="" Nov 03 20:28:48.527969 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527842 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b013faae-082b-495b-9d77-b4371e82e379" volumeName="kubernetes.io/projected/b013faae-082b-495b-9d77-b4371e82e379-kube-api-access-szxvh" seLinuxMountContext="" Nov 03 20:28:48.527969 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527855 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e65f973-499a-4dac-af00-b485fb5dbbfe" volumeName="kubernetes.io/projected/7e65f973-499a-4dac-af00-b485fb5dbbfe-kube-api-access-c6ls8" seLinuxMountContext="" Nov 03 20:28:48.527969 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527873 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5474343-621e-4dd9-948c-ad71bd6f00c2" volumeName="kubernetes.io/projected/c5474343-621e-4dd9-948c-ad71bd6f00c2-kube-api-access-s62xs" seLinuxMountContext="" Nov 03 20:28:48.527969 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527884 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d48c0b65-db57-43b2-8e7c-6e9e8feddf89" volumeName="kubernetes.io/projected/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-kube-api-access-bmnw9" seLinuxMountContext="" Nov 03 20:28:48.527969 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527898 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d48c0b65-db57-43b2-8e7c-6e9e8feddf89" volumeName="kubernetes.io/secret/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 20:28:48.527969 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527908 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5125fef-6a2d-45d6-99d2-90915b575f32" volumeName="kubernetes.io/configmap/d5125fef-6a2d-45d6-99d2-90915b575f32-cni-binary-copy" seLinuxMountContext="" Nov 03 20:28:48.527969 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527920 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7ab819a-4321-4958-a5d5-230cbdbd9428" volumeName="kubernetes.io/configmap/f7ab819a-4321-4958-a5d5-230cbdbd9428-serviceca" seLinuxMountContext="" Nov 03 20:28:48.527969 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527933 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7ab819a-4321-4958-a5d5-230cbdbd9428" volumeName="kubernetes.io/projected/f7ab819a-4321-4958-a5d5-230cbdbd9428-kube-api-access-8ss5t" seLinuxMountContext="" Nov 03 20:28:48.527969 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527946 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3514724f-deea-4b4e-bce9-48ce7862d6f0" volumeName="kubernetes.io/projected/3514724f-deea-4b4e-bce9-48ce7862d6f0-kube-api-access-hwv4c" seLinuxMountContext="" Nov 03 20:28:48.527969 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527957 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5474343-621e-4dd9-948c-ad71bd6f00c2" volumeName="kubernetes.io/empty-dir/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-tuned" seLinuxMountContext="" Nov 03 20:28:48.527969 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527966 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d48c0b65-db57-43b2-8e7c-6e9e8feddf89" volumeName="kubernetes.io/configmap/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-ovnkube-script-lib" seLinuxMountContext="" Nov 03 20:28:48.528759 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527985 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d48c0b65-db57-43b2-8e7c-6e9e8feddf89" volumeName="kubernetes.io/configmap/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-env-overrides" seLinuxMountContext="" Nov 03 20:28:48.528759 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527994 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e99c666f-7bb3-4856-a2fc-935ee184dae3" volumeName="kubernetes.io/configmap/e99c666f-7bb3-4856-a2fc-935ee184dae3-konnectivity-ca" seLinuxMountContext="" Nov 03 20:28:48.528759 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.527732 4493 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 20:28:48.528759 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.528021 4493 factory.go:55] Registering systemd factory Nov 03 20:28:48.528759 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.528036 4493 factory.go:223] Registration of the systemd container factory successfully Nov 03 20:28:48.528759 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.528051 4493 factory.go:103] Registering Raw factory Nov 03 20:28:48.528759 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.528060 4493 manager.go:1199] Started watching for new ooms in manager Nov 03 20:28:48.528759 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.528002 4493 reconstruct.go:97] "Volume reconstruction finished" Nov 03 20:28:48.528759 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.528078 4493 reconciler.go:29] "Reconciler: start to sync state" Nov 03 20:28:48.528759 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.528357 4493 manager.go:319] Starting recovery of all containers Nov 03 20:28:48.534954 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.534931 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 20:28:48.537644 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.537604 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 20:28:48.552240 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.552208 4493 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 20:28:48.553944 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.553878 4493 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 20:28:48.553944 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.553904 4493 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 20:28:48.553944 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.553922 4493 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 20:28:48.553944 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:48.553955 4493 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 20:28:48.562557 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.562529 4493 generic.go:356] "Generic (PLEG): container finished" podID="9844c086-efc5-46f3-9d37-34c2fa013f34" containerID="9d723a4cbb2ec2ff9d6f41b1d7223a232a89682015e31006a3e04aef8a852958" exitCode=0 Nov 03 20:28:48.562557 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.562556 4493 generic.go:356] "Generic (PLEG): container finished" podID="9844c086-efc5-46f3-9d37-34c2fa013f34" containerID="ed25415b299e79f7d1814992764e8f11bb63ba3d2245bbf6f036877f7aeeac48" exitCode=0 Nov 03 20:28:48.562732 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.562566 4493 generic.go:356] "Generic (PLEG): container finished" podID="9844c086-efc5-46f3-9d37-34c2fa013f34" containerID="c446a9d0a01f9762533143bfcdfc18e2bda5724ac90326436d1033ba7a4c7cbc" exitCode=0 Nov 03 20:28:48.562732 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.562572 4493 generic.go:356] "Generic (PLEG): container finished" podID="9844c086-efc5-46f3-9d37-34c2fa013f34" containerID="e5b5ef8d438b876180c8e668c3f31f93e4270e40fb5ea9e20ae5190351cc583a" exitCode=0 Nov 03 20:28:48.562732 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.562577 4493 generic.go:356] "Generic (PLEG): container finished" podID="9844c086-efc5-46f3-9d37-34c2fa013f34" containerID="c82a8cac5d718b43a49f3d4fea60746ec0665422149c90cf621ede8d693a1550" exitCode=0 Nov 03 20:28:48.562732 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.562582 4493 generic.go:356] "Generic (PLEG): container finished" podID="9844c086-efc5-46f3-9d37-34c2fa013f34" containerID="fa97359fa56a27ebfe28bbb6de377e7daa3bc7bb815d3369a38c01424ee04d6a" exitCode=0 Nov 03 20:28:48.572891 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.572870 4493 generic.go:356] "Generic (PLEG): container finished" podID="cb4d50458eb702ff22ca2222c359eed8" containerID="5615c1bf4299b681a28cedf9ab1bec91cd637adea9348e9fb47e1a89edc518f1" exitCode=0 Nov 03 20:28:48.591777 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.591758 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 20:28:48.592102 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.592067 4493 generic.go:356] "Generic (PLEG): container finished" podID="d48c0b65-db57-43b2-8e7c-6e9e8feddf89" containerID="3dc8c3593cdfb1a0bf12dd431bd4ec5b9a97d901e47b7bd8ca0ce2b8112d3102" exitCode=1 Nov 03 20:28:48.598296 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.598277 4493 manager.go:324] Recovery completed Nov 03 20:28:48.612970 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.612889 4493 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:48.614051 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.614030 4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:48.614140 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.614061 4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:48.614140 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.614071 4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:48.616131 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.616115 4493 cpu_manager.go:226] "Starting" policy="none" Nov 03 20:28:48.616131 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.616124 4493 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:48.616258 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.616127 4493 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 20:28:48.616258 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.616209 4493 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 20:28:48.616444 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.616426 4493 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 20:28:48.616507 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.616445 4493 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 20:28:48.616567 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.616549 4493 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 20:28:48.616567 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.616560 4493 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 20:28:48.616653 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.616569 4493 policy_none.go:50] "Start" Nov 03 20:28:48.616653 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.616585 4493 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 20:28:48.616653 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.616600 4493 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 20:28:48.616804 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.616788 4493 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 20:28:48.616869 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.616806 4493 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 20:28:48.616869 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.616814 4493 policy_none.go:44] "Start" Nov 03 20:28:48.616960 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.616946 4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:48.617004 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.616981 4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:48.617004 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.616996 4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:48.620799 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.620776 4493 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 20:28:48.624407 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.624395 4493 manager.go:341] "Starting Device Plugin manager" Nov 03 20:28:48.624480 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:48.624419 4493 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 20:28:48.624480 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.624428 4493 server.go:87] "Starting device plugin registration server" Nov 03 20:28:48.624674 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.624664 4493 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 20:28:48.624709 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.624676 4493 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 20:28:48.624804 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.624787 4493 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 20:28:48.625210 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.625171 4493 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 20:28:48.625295 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.625241 4493 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 20:28:48.625295 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.625249 4493 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 20:28:48.625414 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.625312 4493 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:28:48.625206819Z","UUID":"c8faaaed-171a-4811-bb42-4100df1e8bca","Handler":null,"Name":"","Endpoint":""} Nov 03 20:28:48.626370 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:48.626348 4493 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 20:28:48.626480 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:48.626394 4493 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-26.us-west-2.compute.internal\" not found" Nov 03 20:28:48.626794 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.626778 4493 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 20:28:48.626900 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.626804 4493 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 20:28:48.655645 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.655035 4493 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal"] Nov 03 20:28:48.655795 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.655724 4493 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:48.656639 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.656615 4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:48.656725 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.656648 4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:48.656725 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.656672 4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:48.656931 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.656921 4493 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:48.657060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.657047 4493 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:48.657699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.657681 4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:48.657786 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.657699 4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:48.657786 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.657710 4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:48.657786 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.657721 4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:48.657786 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.657724 4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:48.657786 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.657741 4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:48.658008 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:48.657932 4493 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-26.us-west-2.compute.internal\" not found" node="ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:48.658008 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.657946 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal" event={"ID":"cb4d50458eb702ff22ca2222c359eed8","Type":"ContainerStarted","Data":"6098514fcedcaa160b8cd7b6a3872d6cca8720c3bd35197a99a6f08b4b90619c"} Nov 03 20:28:48.658008 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.657981 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal" event={"ID":"cb4d50458eb702ff22ca2222c359eed8","Type":"ContainerDied","Data":"5615c1bf4299b681a28cedf9ab1bec91cd637adea9348e9fb47e1a89edc518f1"} Nov 03 20:28:48.658008 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.657996 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal" event={"ID":"cb4d50458eb702ff22ca2222c359eed8","Type":"ContainerStarted","Data":"b0742edfa33aa6021b9dc3a08e45c023e436ce141fd7b15572b3831f1a5a7c66"} Nov 03 20:28:48.658126 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.658013 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal" event={"ID":"733c8a62db18973abc6d8fb96bb89db1","Type":"ContainerStarted","Data":"0a105902a4f8ede40be45e8206e2bf89c53193a0c115daa302d8aacb7729dacb"} Nov 03 20:28:48.658126 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.658033 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal" event={"ID":"733c8a62db18973abc6d8fb96bb89db1","Type":"ContainerStarted","Data":"0c5b6a1b1f30d3e9122ef65b5a92319eaecd4674ea7acfb1194edbbb5ea43818"} Nov 03 20:28:48.658126 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.658046 4493 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:48.658750 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.658736 4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:48.658798 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.658761 4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:48.658798 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.658771 4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:48.658927 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:48.658915 4493 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-26.us-west-2.compute.internal\" not found" node="ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:48.704595 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.704566 4493 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:48.704950 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.704923 4493 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:48.705036 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.705021 4493 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 20:28:48.705036 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.705028 4493 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:48.725201 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.725176 4493 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:48.726209 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.726188 4493 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:48.726367 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.726192 4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:48.726367 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.726309 4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:48.726367 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.726326 4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:48.726513 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.726394 4493 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:48.752303 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.752270 4493 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:48.797012 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.796979 4493 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:28:48.797167 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:48.797037 4493 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:48.797167 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.797053 4493 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:48.805789 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.805762 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/733c8a62db18973abc6d8fb96bb89db1-config\") pod \"kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal\" (UID: \"733c8a62db18973abc6d8fb96bb89db1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:48.805789 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.805788 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cb4d50458eb702ff22ca2222c359eed8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal\" (UID: \"cb4d50458eb702ff22ca2222c359eed8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:48.805921 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.805816 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb4d50458eb702ff22ca2222c359eed8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal\" (UID: \"cb4d50458eb702ff22ca2222c359eed8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:48.830862 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.830834 4493 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:48.830984 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.830903 4493 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:48.831524 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.831510 4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeNotReady" Nov 03 20:28:48.831593 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.831548 4493 setters.go:546] "Node became not ready" node="ip-10-0-138-26.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T20:28:48Z","lastTransitionTime":"2026-11-03T20:28:48Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 03 20:28:48.855849 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.855819 4493 nodeinfomanager.go:471] existing CSINode "ip-10-0-138-26.us-west-2.compute.internal" is owned by different node (oldNodeID="cb2b041e-69b2-43bb-888d-027b7049dc8e", newNodeID=""), cleaning up... Nov 03 20:28:48.883458 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.883398 4493 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:28:48.883458 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:48.883450 4493 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:48.906880 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.906852 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cb4d50458eb702ff22ca2222c359eed8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal\" (UID: \"cb4d50458eb702ff22ca2222c359eed8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:48.906880 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.906881 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb4d50458eb702ff22ca2222c359eed8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal\" (UID: \"cb4d50458eb702ff22ca2222c359eed8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:48.907014 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.906901 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/733c8a62db18973abc6d8fb96bb89db1-config\") pod \"kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal\" (UID: \"733c8a62db18973abc6d8fb96bb89db1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:48.907014 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.906954 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/733c8a62db18973abc6d8fb96bb89db1-config\") pod \"kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal\" (UID: \"733c8a62db18973abc6d8fb96bb89db1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:48.907014 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.906967 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb4d50458eb702ff22ca2222c359eed8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal\" (UID: \"cb4d50458eb702ff22ca2222c359eed8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:48.907014 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:48.906954 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cb4d50458eb702ff22ca2222c359eed8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal\" (UID: \"cb4d50458eb702ff22ca2222c359eed8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:49.509183 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.509145 4493 apiserver.go:52] "Watching apiserver" Nov 03 20:28:49.595934 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.595907 4493 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:49.596082 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.596044 4493 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:49.607166 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.607146 4493 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 20:28:49.607609 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.607590 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-ccxwr","openshift-multus/multus-pv8vt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-tl7xc","kube-system/kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal","openshift-dns/dns-default-jjqxd","kube-system/global-pull-secret-syncer-2sjv9","openshift-image-registry/node-ca-5tx7t","openshift-network-operator/iptables-alerter-bjbkk","openshift-ingress-canary/ingress-canary-5jfs9","openshift-insights/periodic-gathering-rr8h2-bjl4n","openshift-kube-storage-version-migrator/migrator-66df5d496b-gj9nc","openshift-network-diagnostics/network-check-target-l4pbv","openshift-cluster-node-tuning-operator/tuned-bv4jb","openshift-dns/node-resolver-kzsn2","openshift-insights/insights-runtime-extractor-67nxb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26","openshift-multus/network-metrics-daemon-f28pp","openshift-ovn-kubernetes/ovnkube-node-82vml"] Nov 03 20:28:49.613800 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.613777 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:49.614702 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.614685 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:49.615394 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.615378 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:49.627208 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.627183 4493 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 20:28:49.653611 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.653593 4493 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:28:49.653697 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:49.653644 4493 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:49.653849 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.653837 4493 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:28:49.653881 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:49.653865 4493 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-26.us-west-2.compute.internal" Nov 03 20:28:49.657115 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.657098 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jxjwz\"" Nov 03 20:28:49.657312 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.657298 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 20:28:49.659885 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.659870 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pxhx8\"" Nov 03 20:28:49.710226 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.710203 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6ls8\" (UniqueName: \"kubernetes.io/projected/7e65f973-499a-4dac-af00-b485fb5dbbfe-kube-api-access-c6ls8\") pod \"ingress-canary-5jfs9\" (UID: \"7e65f973-499a-4dac-af00-b485fb5dbbfe\") " pod="openshift-ingress-canary/ingress-canary-5jfs9" Nov 03 20:28:49.710325 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.710231 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b013faae-082b-495b-9d77-b4371e82e379-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:49.710325 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.710249 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-etc-kubernetes\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.710325 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.710266 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrlqh\" (UniqueName: \"kubernetes.io/projected/d5125fef-6a2d-45d6-99d2-90915b575f32-kube-api-access-hrlqh\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.710325 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.710283 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9844c086-efc5-46f3-9d37-34c2fa013f34-cni-binary-copy\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:49.710534 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.710363 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-lib-modules\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.710534 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.710479 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef30d813-b7ed-417d-a9fe-85118bf77a95-kubelet-config\") pod \"global-pull-secret-syncer-2sjv9\" (UID: \"ef30d813-b7ed-417d-a9fe-85118bf77a95\") " pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:49.710672 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.710561 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szxvh\" (UniqueName: \"kubernetes.io/projected/b013faae-082b-495b-9d77-b4371e82e379-kube-api-access-szxvh\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:49.710672 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.710599 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9844c086-efc5-46f3-9d37-34c2fa013f34-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:49.710672 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.710623 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-sysctl-d\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.710672 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.710651 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-sysctl-conf\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.711007 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.710680 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nlpn\" (UniqueName: \"kubernetes.io/projected/6cd2ab07-4196-4ed1-b357-38f01b99b71d-kube-api-access-6nlpn\") pod \"network-check-target-l4pbv\" (UID: \"6cd2ab07-4196-4ed1-b357-38f01b99b71d\") " pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:49.711007 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.710705 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfrc5\" (UniqueName: \"kubernetes.io/projected/59643abf-923a-4d4d-8235-c2ad22c8aea0-kube-api-access-wfrc5\") pod \"node-resolver-kzsn2\" (UID: \"59643abf-923a-4d4d-8235-c2ad22c8aea0\") " pod="openshift-dns/node-resolver-kzsn2" Nov 03 20:28:49.711007 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.710740 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-systemd-units\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.711007 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.710792 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-run-netns\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.711007 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.710894 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f7ab819a-4321-4958-a5d5-230cbdbd9428-serviceca\") pod \"node-ca-5tx7t\" (UID: \"f7ab819a-4321-4958-a5d5-230cbdbd9428\") " pod="openshift-image-registry/node-ca-5tx7t" Nov 03 20:28:49.711007 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.710940 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9844c086-efc5-46f3-9d37-34c2fa013f34-system-cni-dir\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:49.711007 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.710997 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-sysconfig\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.711240 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.711029 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-var-lib-kubelet\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.711240 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.711058 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e99c666f-7bb3-4856-a2fc-935ee184dae3-agent-certs\") pod \"konnectivity-agent-ccxwr\" (UID: \"e99c666f-7bb3-4856-a2fc-935ee184dae3\") " pod="kube-system/konnectivity-agent-ccxwr" Nov 03 20:28:49.711240 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.711087 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-node-log\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.711240 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.711103 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-log-socket\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.711240 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.711142 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-run-ovn-kubernetes\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.711240 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.711202 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:49.711240 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.711231 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b013faae-082b-495b-9d77-b4371e82e379-crio-socket\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:49.711571 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.711276 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:49.711571 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.711307 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-run-netns\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.711571 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.711355 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9487d\" (UniqueName: \"kubernetes.io/projected/2281c3da-747c-4a37-89ec-b764d8a95405-kube-api-access-9487d\") pod \"network-metrics-daemon-f28pp\" (UID: \"2281c3da-747c-4a37-89ec-b764d8a95405\") " pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:49.711571 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.711385 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59643abf-923a-4d4d-8235-c2ad22c8aea0-hosts-file\") pod \"node-resolver-kzsn2\" (UID: \"59643abf-923a-4d4d-8235-c2ad22c8aea0\") " pod="openshift-dns/node-resolver-kzsn2" Nov 03 20:28:49.711571 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.711482 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59643abf-923a-4d4d-8235-c2ad22c8aea0-tmp-dir\") pod \"node-resolver-kzsn2\" (UID: \"59643abf-923a-4d4d-8235-c2ad22c8aea0\") " pod="openshift-dns/node-resolver-kzsn2" Nov 03 20:28:49.711571 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.711503 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-etc-openvswitch\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.711571 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.711519 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-modprobe-d\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.711571 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.711535 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e99c666f-7bb3-4856-a2fc-935ee184dae3-konnectivity-ca\") pod \"konnectivity-agent-ccxwr\" (UID: \"e99c666f-7bb3-4856-a2fc-935ee184dae3\") " pod="kube-system/konnectivity-agent-ccxwr" Nov 03 20:28:49.711571 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.711578 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59643abf-923a-4d4d-8235-c2ad22c8aea0-tmp-dir\") pod \"node-resolver-kzsn2\" (UID: \"59643abf-923a-4d4d-8235-c2ad22c8aea0\") " pod="openshift-dns/node-resolver-kzsn2" Nov 03 20:28:49.711896 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.711610 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdhpf\" (UniqueName: \"kubernetes.io/projected/4cc66ac6-6311-4144-8db1-3c39182e1a19-kube-api-access-vdhpf\") pod \"iptables-alerter-bjbkk\" (UID: \"4cc66ac6-6311-4144-8db1-3c39182e1a19\") " pod="openshift-network-operator/iptables-alerter-bjbkk" Nov 03 20:28:49.711896 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.711636 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-run-openvswitch\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.711896 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.711660 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-env-overrides\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.711896 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.711676 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e65f973-499a-4dac-af00-b485fb5dbbfe-cert\") pod \"ingress-canary-5jfs9\" (UID: \"7e65f973-499a-4dac-af00-b485fb5dbbfe\") " pod="openshift-ingress-canary/ingress-canary-5jfs9" Nov 03 20:28:49.711896 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.711691 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-system-cni-dir\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.711896 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.711706 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-run-k8s-cni-cncf-io\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.712219 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.712200 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs\") pod \"network-metrics-daemon-f28pp\" (UID: \"2281c3da-747c-4a37-89ec-b764d8a95405\") " pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:49.712273 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.712239 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9qrh\" (UniqueName: \"kubernetes.io/projected/34b6b405-a677-4fef-8ee6-37d84a1c5902-kube-api-access-h9qrh\") pod \"periodic-gathering-rr8h2-bjl4n\" (UID: \"34b6b405-a677-4fef-8ee6-37d84a1c5902\") " pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" Nov 03 20:28:49.712327 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.712267 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-sys-fs\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:49.712327 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.712297 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5125fef-6a2d-45d6-99d2-90915b575f32-cni-binary-copy\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.712442 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.712359 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9844c086-efc5-46f3-9d37-34c2fa013f34-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:49.712442 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.712389 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-host\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.712442 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.712416 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-run-systemd\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.712604 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.712445 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-ovnkube-config\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.712604 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.712471 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-registration-dir\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:49.712604 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.712519 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-etc-selinux\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:49.712604 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.712581 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b013faae-082b-495b-9d77-b4371e82e379-data-volume\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:49.712738 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.712619 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-var-lib-cni-multus\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.712738 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.712653 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5125fef-6a2d-45d6-99d2-90915b575f32-multus-daemon-config\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.712738 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.712680 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b013faae-082b-495b-9d77-b4371e82e379-data-volume\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:49.712738 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.712699 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:49.712738 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.712717 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrs5m\" (UniqueName: \"kubernetes.io/projected/9844c086-efc5-46f3-9d37-34c2fa013f34-kube-api-access-zrs5m\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:49.712738 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.712732 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-cni-bin\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.712978 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.712754 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef30d813-b7ed-417d-a9fe-85118bf77a95-original-pull-secret\") pod \"global-pull-secret-syncer-2sjv9\" (UID: \"ef30d813-b7ed-417d-a9fe-85118bf77a95\") " pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:49.712978 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.712770 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34b6b405-a677-4fef-8ee6-37d84a1c5902-service-ca-bundle\") pod \"periodic-gathering-rr8h2-bjl4n\" (UID: \"34b6b405-a677-4fef-8ee6-37d84a1c5902\") " pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" Nov 03 20:28:49.712978 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.712790 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-socket-dir\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:49.712978 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.712815 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fhqp\" (UniqueName: \"kubernetes.io/projected/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-kube-api-access-4fhqp\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:49.712978 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.712855 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-run-multus-certs\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.713208 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.712976 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-systemd\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.713208 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713012 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-run\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.713208 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713036 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/34b6b405-a677-4fef-8ee6-37d84a1c5902-archives-path\") pod \"periodic-gathering-rr8h2-bjl4n\" (UID: \"34b6b405-a677-4fef-8ee6-37d84a1c5902\") " pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" Nov 03 20:28:49.713208 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713102 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/34b6b405-a677-4fef-8ee6-37d84a1c5902-archives-path\") pod \"periodic-gathering-rr8h2-bjl4n\" (UID: \"34b6b405-a677-4fef-8ee6-37d84a1c5902\") " pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" Nov 03 20:28:49.713208 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713128 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwv4c\" (UniqueName: \"kubernetes.io/projected/3514724f-deea-4b4e-bce9-48ce7862d6f0-kube-api-access-hwv4c\") pod \"migrator-66df5d496b-gj9nc\" (UID: \"3514724f-deea-4b4e-bce9-48ce7862d6f0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gj9nc" Nov 03 20:28:49.713208 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713152 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-var-lib-openvswitch\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.713208 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713168 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-multus-socket-dir-parent\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.713565 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713246 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/480d6b9d-4f58-4779-a2cb-aa4b64f93881-config-volume\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:49.713565 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713278 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/480d6b9d-4f58-4779-a2cb-aa4b64f93881-tmp-dir\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:49.713565 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713294 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5474343-621e-4dd9-948c-ad71bd6f00c2-tmp\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.713565 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713363 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5474343-621e-4dd9-948c-ad71bd6f00c2-tmp\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.713565 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713389 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-slash\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.713565 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713401 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/480d6b9d-4f58-4779-a2cb-aa4b64f93881-tmp-dir\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:49.713565 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713414 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.713565 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713433 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-ovnkube-script-lib\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.713565 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713458 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-var-lib-cni-bin\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.713565 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713474 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef30d813-b7ed-417d-a9fe-85118bf77a95-dbus\") pod \"global-pull-secret-syncer-2sjv9\" (UID: \"ef30d813-b7ed-417d-a9fe-85118bf77a95\") " pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:49.713565 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713492 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-kubelet\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.713565 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713538 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmnw9\" (UniqueName: \"kubernetes.io/projected/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-kube-api-access-bmnw9\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.714054 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713582 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-hostroot\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.714054 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713628 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9844c086-efc5-46f3-9d37-34c2fa013f34-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:49.714054 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713657 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4cc66ac6-6311-4144-8db1-3c39182e1a19-iptables-alerter-script\") pod \"iptables-alerter-bjbkk\" (UID: \"4cc66ac6-6311-4144-8db1-3c39182e1a19\") " pod="openshift-network-operator/iptables-alerter-bjbkk" Nov 03 20:28:49.714054 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713697 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34b6b405-a677-4fef-8ee6-37d84a1c5902-trusted-ca-bundle\") pod \"periodic-gathering-rr8h2-bjl4n\" (UID: \"34b6b405-a677-4fef-8ee6-37d84a1c5902\") " pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" Nov 03 20:28:49.714054 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713728 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-ovn-node-metrics-cert\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.714054 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713787 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-multus-cni-dir\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.714054 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713810 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7ab819a-4321-4958-a5d5-230cbdbd9428-host\") pod \"node-ca-5tx7t\" (UID: \"f7ab819a-4321-4958-a5d5-230cbdbd9428\") " pod="openshift-image-registry/node-ca-5tx7t" Nov 03 20:28:49.714054 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713828 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-var-lib-kubelet\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.714054 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713880 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-multus-conf-dir\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.714054 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713907 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m49g\" (UniqueName: \"kubernetes.io/projected/480d6b9d-4f58-4779-a2cb-aa4b64f93881-kube-api-access-2m49g\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:49.714054 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713932 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-kubernetes\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.714054 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.713955 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-sys\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.714054 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.714008 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-tuned\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.714054 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.714040 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ss5t\" (UniqueName: \"kubernetes.io/projected/f7ab819a-4321-4958-a5d5-230cbdbd9428-kube-api-access-8ss5t\") pod \"node-ca-5tx7t\" (UID: \"f7ab819a-4321-4958-a5d5-230cbdbd9428\") " pod="openshift-image-registry/node-ca-5tx7t" Nov 03 20:28:49.714585 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.714074 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-device-dir\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:49.714585 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.714150 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-cnibin\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.714585 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.714188 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9844c086-efc5-46f3-9d37-34c2fa013f34-cnibin\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:49.714585 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.714204 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-tuned\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.714585 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.714220 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s62xs\" (UniqueName: \"kubernetes.io/projected/c5474343-621e-4dd9-948c-ad71bd6f00c2-kube-api-access-s62xs\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.714585 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.714237 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cc66ac6-6311-4144-8db1-3c39182e1a19-host-slash\") pod \"iptables-alerter-bjbkk\" (UID: \"4cc66ac6-6311-4144-8db1-3c39182e1a19\") " pod="openshift-network-operator/iptables-alerter-bjbkk" Nov 03 20:28:49.714585 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.714291 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-run-ovn\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.714585 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.714314 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-cni-netd\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.758062 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.758039 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 20:28:49.758746 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.758715 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 20:28:49.758931 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.758761 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 20:28:49.758931 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.758779 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 20:28:49.759158 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.759100 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 20:28:49.759511 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.759426 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 20:28:49.759672 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.759623 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:28:49.759672 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.759653 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 20:28:49.759895 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.759686 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 20:28:49.759895 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.759687 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 20:28:49.759895 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.759721 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 20:28:49.759895 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.759736 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 20:28:49.759895 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.759768 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 20:28:49.759895 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.759789 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pqh8h\"" Nov 03 20:28:49.759895 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.759842 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 20:28:49.760402 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.759958 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 20:28:49.760402 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.760029 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:28:49.760402 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.760052 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8pktm\"" Nov 03 20:28:49.760402 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.760223 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 20:28:49.760402 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.760230 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 20:28:49.760402 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.760258 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 20:28:49.760402 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.760317 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 20:28:49.760402 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.760324 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jb4cj\"" Nov 03 20:28:49.760402 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.760363 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 20:28:49.760402 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.760389 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 20:28:49.760924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.760436 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xvpcc\"" Nov 03 20:28:49.760924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.760463 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 20:28:49.760924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.760492 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 20:28:49.760924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.760570 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 20:28:49.760924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.760608 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 20:28:49.760924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.760616 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 20:28:49.760924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.760641 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-d54s8\"" Nov 03 20:28:49.760924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.760636 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 20:28:49.760924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.760666 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 20:28:49.760924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.760677 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 20:28:49.760924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.760709 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 20:28:49.760924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.760760 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w6582\"" Nov 03 20:28:49.760924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.760848 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9844c086-efc5-46f3-9d37-34c2fa013f34-cni-binary-copy\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:49.761591 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.761244 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 20:28:49.761591 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.761288 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f7ab819a-4321-4958-a5d5-230cbdbd9428-serviceca\") pod \"node-ca-5tx7t\" (UID: \"f7ab819a-4321-4958-a5d5-230cbdbd9428\") " pod="openshift-image-registry/node-ca-5tx7t" Nov 03 20:28:49.761591 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.761276 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 20:28:49.761591 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.761351 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kpkvl\"" Nov 03 20:28:49.761591 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.761428 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ntq9n\"" Nov 03 20:28:49.761591 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.761442 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 20:28:49.761591 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.761487 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 20:28:49.761591 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.761527 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-fpmzx\"" Nov 03 20:28:49.761591 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.761543 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 20:28:49.761863 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.761634 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mlgd9\"" Nov 03 20:28:49.761863 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.761641 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 20:28:49.761863 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.761669 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e99c666f-7bb3-4856-a2fc-935ee184dae3-agent-certs\") pod \"konnectivity-agent-ccxwr\" (UID: \"e99c666f-7bb3-4856-a2fc-935ee184dae3\") " pod="kube-system/konnectivity-agent-ccxwr" Nov 03 20:28:49.762234 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.762219 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 20:28:49.762325 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.762274 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w48hg\"" Nov 03 20:28:49.762325 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.762283 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 20:28:49.762325 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.762321 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-w9qmc\"" Nov 03 20:28:49.762471 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.762374 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-env-overrides\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.762471 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.762394 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 20:28:49.762471 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.762423 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 20:28:49.762572 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.762469 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h9ssl\"" Nov 03 20:28:49.762572 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.762520 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 20:28:49.762634 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.762618 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5125fef-6a2d-45d6-99d2-90915b575f32-cni-binary-copy\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.762662 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.762615 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e65f973-499a-4dac-af00-b485fb5dbbfe-cert\") pod \"ingress-canary-5jfs9\" (UID: \"7e65f973-499a-4dac-af00-b485fb5dbbfe\") " pod="openshift-ingress-canary/ingress-canary-5jfs9" Nov 03 20:28:49.762662 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.762620 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9844c086-efc5-46f3-9d37-34c2fa013f34-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:49.762769 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.762756 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-ovnkube-config\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.762895 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.762878 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5125fef-6a2d-45d6-99d2-90915b575f32-multus-daemon-config\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.762959 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.762946 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34b6b405-a677-4fef-8ee6-37d84a1c5902-service-ca-bundle\") pod \"periodic-gathering-rr8h2-bjl4n\" (UID: \"34b6b405-a677-4fef-8ee6-37d84a1c5902\") " pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" Nov 03 20:28:49.763056 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.763044 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef30d813-b7ed-417d-a9fe-85118bf77a95-original-pull-secret\") pod \"global-pull-secret-syncer-2sjv9\" (UID: \"ef30d813-b7ed-417d-a9fe-85118bf77a95\") " pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:49.763829 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.763790 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/480d6b9d-4f58-4779-a2cb-aa4b64f93881-config-volume\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:49.763944 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.763919 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4cc66ac6-6311-4144-8db1-3c39182e1a19-iptables-alerter-script\") pod \"iptables-alerter-bjbkk\" (UID: \"4cc66ac6-6311-4144-8db1-3c39182e1a19\") " pod="openshift-network-operator/iptables-alerter-bjbkk" Nov 03 20:28:49.763995 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.763953 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9844c086-efc5-46f3-9d37-34c2fa013f34-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:49.764032 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.764014 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-ovn-node-metrics-cert\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.764224 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.764211 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-ovnkube-script-lib\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.772240 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.772224 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e99c666f-7bb3-4856-a2fc-935ee184dae3-konnectivity-ca\") pod \"konnectivity-agent-ccxwr\" (UID: \"e99c666f-7bb3-4856-a2fc-935ee184dae3\") " pod="kube-system/konnectivity-agent-ccxwr" Nov 03 20:28:49.805102 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.805083 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c6vtd\"" Nov 03 20:28:49.805503 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.805491 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8hb94\"" Nov 03 20:28:49.807151 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.807078 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 20:28:49.808095 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.807903 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szxvh\" (UniqueName: \"kubernetes.io/projected/b013faae-082b-495b-9d77-b4371e82e379-kube-api-access-szxvh\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:49.808095 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.807915 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrlqh\" (UniqueName: \"kubernetes.io/projected/d5125fef-6a2d-45d6-99d2-90915b575f32-kube-api-access-hrlqh\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.808258 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.808199 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6ls8\" (UniqueName: \"kubernetes.io/projected/7e65f973-499a-4dac-af00-b485fb5dbbfe-kube-api-access-c6ls8\") pod \"ingress-canary-5jfs9\" (UID: \"7e65f973-499a-4dac-af00-b485fb5dbbfe\") " pod="openshift-ingress-canary/ingress-canary-5jfs9" Nov 03 20:28:49.809422 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.808454 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdhpf\" (UniqueName: \"kubernetes.io/projected/4cc66ac6-6311-4144-8db1-3c39182e1a19-kube-api-access-vdhpf\") pod \"iptables-alerter-bjbkk\" (UID: \"4cc66ac6-6311-4144-8db1-3c39182e1a19\") " pod="openshift-network-operator/iptables-alerter-bjbkk" Nov 03 20:28:49.809422 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.808486 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9487d\" (UniqueName: \"kubernetes.io/projected/2281c3da-747c-4a37-89ec-b764d8a95405-kube-api-access-9487d\") pod \"network-metrics-daemon-f28pp\" (UID: \"2281c3da-747c-4a37-89ec-b764d8a95405\") " pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:49.809422 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.808417 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nlpn\" (UniqueName: \"kubernetes.io/projected/6cd2ab07-4196-4ed1-b357-38f01b99b71d-kube-api-access-6nlpn\") pod \"network-check-target-l4pbv\" (UID: \"6cd2ab07-4196-4ed1-b357-38f01b99b71d\") " pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:49.809422 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.808936 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fhqp\" (UniqueName: \"kubernetes.io/projected/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-kube-api-access-4fhqp\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:49.809422 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.808974 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m49g\" (UniqueName: \"kubernetes.io/projected/480d6b9d-4f58-4779-a2cb-aa4b64f93881-kube-api-access-2m49g\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:49.809422 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.809103 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmnw9\" (UniqueName: \"kubernetes.io/projected/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-kube-api-access-bmnw9\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.809422 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.809327 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ss5t\" (UniqueName: \"kubernetes.io/projected/f7ab819a-4321-4958-a5d5-230cbdbd9428-kube-api-access-8ss5t\") pod \"node-ca-5tx7t\" (UID: \"f7ab819a-4321-4958-a5d5-230cbdbd9428\") " pod="openshift-image-registry/node-ca-5tx7t" Nov 03 20:28:49.809856 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.809826 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfrc5\" (UniqueName: \"kubernetes.io/projected/59643abf-923a-4d4d-8235-c2ad22c8aea0-kube-api-access-wfrc5\") pod \"node-resolver-kzsn2\" (UID: \"59643abf-923a-4d4d-8235-c2ad22c8aea0\") " pod="openshift-dns/node-resolver-kzsn2" Nov 03 20:28:49.809924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.809839 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9qrh\" (UniqueName: \"kubernetes.io/projected/34b6b405-a677-4fef-8ee6-37d84a1c5902-kube-api-access-h9qrh\") pod \"periodic-gathering-rr8h2-bjl4n\" (UID: \"34b6b405-a677-4fef-8ee6-37d84a1c5902\") " pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" Nov 03 20:28:49.809924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.809879 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrs5m\" (UniqueName: \"kubernetes.io/projected/9844c086-efc5-46f3-9d37-34c2fa013f34-kube-api-access-zrs5m\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:49.809924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.809882 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwv4c\" (UniqueName: \"kubernetes.io/projected/3514724f-deea-4b4e-bce9-48ce7862d6f0-kube-api-access-hwv4c\") pod \"migrator-66df5d496b-gj9nc\" (UID: \"3514724f-deea-4b4e-bce9-48ce7862d6f0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gj9nc" Nov 03 20:28:49.811394 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.810090 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s62xs\" (UniqueName: \"kubernetes.io/projected/c5474343-621e-4dd9-948c-ad71bd6f00c2-kube-api-access-s62xs\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.814212 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814193 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34b6b405-a677-4fef-8ee6-37d84a1c5902-trusted-ca-bundle\") pod \"periodic-gathering-rr8h2-bjl4n\" (UID: \"34b6b405-a677-4fef-8ee6-37d84a1c5902\") " pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" Nov 03 20:28:49.814637 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814622 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-var-lib-cni-multus\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.814681 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814645 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:49.814681 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814661 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-cni-bin\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.814681 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814676 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-socket-dir\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:49.814804 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814690 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-run-multus-certs\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.814804 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814703 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-systemd\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.814804 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814717 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-run\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.814804 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814734 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-var-lib-openvswitch\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.814804 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814734 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-cni-bin\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.814804 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814765 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-socket-dir\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:49.814804 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814773 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-systemd\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.814804 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:49.814781 4493 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:28:49.814804 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814786 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-var-lib-cni-multus\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.814804 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814791 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-run-multus-certs\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.814804 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814807 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-multus-socket-dir-parent\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.815128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814819 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-run\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.815128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814771 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-multus-socket-dir-parent\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.815128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814848 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-var-lib-openvswitch\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.815128 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:49.814864 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls podName:480d6b9d-4f58-4779-a2cb-aa4b64f93881 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:50.314847829 +0000 UTC m=+1.872838322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls") pod "dns-default-jjqxd" (UID: "480d6b9d-4f58-4779-a2cb-aa4b64f93881") : secret "dns-default-metrics-tls" not found Nov 03 20:28:49.815128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814885 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-slash\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.815128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814908 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-slash\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.815128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814908 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.815128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814936 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-var-lib-cni-bin\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.815128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814934 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.815128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814956 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef30d813-b7ed-417d-a9fe-85118bf77a95-dbus\") pod \"global-pull-secret-syncer-2sjv9\" (UID: \"ef30d813-b7ed-417d-a9fe-85118bf77a95\") " pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:49.815128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814962 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-var-lib-cni-bin\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.815128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814975 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-kubelet\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.815128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.814990 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-hostroot\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.815128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815005 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-multus-cni-dir\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.815128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815022 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7ab819a-4321-4958-a5d5-230cbdbd9428-host\") pod \"node-ca-5tx7t\" (UID: \"f7ab819a-4321-4958-a5d5-230cbdbd9428\") " pod="openshift-image-registry/node-ca-5tx7t" Nov 03 20:28:49.815128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815026 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-kubelet\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.815128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815041 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef30d813-b7ed-417d-a9fe-85118bf77a95-dbus\") pod \"global-pull-secret-syncer-2sjv9\" (UID: \"ef30d813-b7ed-417d-a9fe-85118bf77a95\") " pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:49.815128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815044 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-var-lib-kubelet\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.815128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815059 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7ab819a-4321-4958-a5d5-230cbdbd9428-host\") pod \"node-ca-5tx7t\" (UID: \"f7ab819a-4321-4958-a5d5-230cbdbd9428\") " pod="openshift-image-registry/node-ca-5tx7t" Nov 03 20:28:49.815128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815074 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-multus-conf-dir\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.815128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815075 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-multus-cni-dir\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.815128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815079 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-var-lib-kubelet\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.815128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815093 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-kubernetes\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.815128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815080 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-hostroot\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.815128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815136 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-kubernetes\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815158 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-multus-conf-dir\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815173 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-sys\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815188 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-device-dir\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815201 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-cnibin\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815224 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9844c086-efc5-46f3-9d37-34c2fa013f34-cnibin\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815238 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cc66ac6-6311-4144-8db1-3c39182e1a19-host-slash\") pod \"iptables-alerter-bjbkk\" (UID: \"4cc66ac6-6311-4144-8db1-3c39182e1a19\") " pod="openshift-network-operator/iptables-alerter-bjbkk" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815242 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-sys\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815256 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-device-dir\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815259 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-run-ovn\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815272 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-cnibin\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815289 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-run-ovn\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815292 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-cni-netd\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815299 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9844c086-efc5-46f3-9d37-34c2fa013f34-cnibin\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815316 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cc66ac6-6311-4144-8db1-3c39182e1a19-host-slash\") pod \"iptables-alerter-bjbkk\" (UID: \"4cc66ac6-6311-4144-8db1-3c39182e1a19\") " pod="openshift-network-operator/iptables-alerter-bjbkk" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815362 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-cni-netd\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815387 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b013faae-082b-495b-9d77-b4371e82e379-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815421 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-etc-kubernetes\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815445 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-lib-modules\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815462 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-etc-kubernetes\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815470 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef30d813-b7ed-417d-a9fe-85118bf77a95-kubelet-config\") pod \"global-pull-secret-syncer-2sjv9\" (UID: \"ef30d813-b7ed-417d-a9fe-85118bf77a95\") " pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815495 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9844c086-efc5-46f3-9d37-34c2fa013f34-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815518 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-sysctl-d\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815535 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-lib-modules\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815550 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-sysctl-conf\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815559 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef30d813-b7ed-417d-a9fe-85118bf77a95-kubelet-config\") pod \"global-pull-secret-syncer-2sjv9\" (UID: \"ef30d813-b7ed-417d-a9fe-85118bf77a95\") " pod="kube-system/global-pull-secret-syncer-2sjv9" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815569 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9844c086-efc5-46f3-9d37-34c2fa013f34-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815574 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-systemd-units\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815606 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-sysctl-d\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815606 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-run-netns\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815614 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-systemd-units\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815629 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-run-netns\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815631 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9844c086-efc5-46f3-9d37-34c2fa013f34-system-cni-dir\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815648 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9844c086-efc5-46f3-9d37-34c2fa013f34-system-cni-dir\") pod \"multus-additional-cni-plugins-tl7xc\" (UID: \"9844c086-efc5-46f3-9d37-34c2fa013f34\") " pod="openshift-multus/multus-additional-cni-plugins-tl7xc" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815660 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-sysconfig\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815676 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-sysctl-conf\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815683 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-var-lib-kubelet\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815704 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-node-log\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815707 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-sysconfig\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815718 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-log-socket\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815732 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-run-ovn-kubernetes\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815738 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-var-lib-kubelet\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815746 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815756 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-node-log\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815761 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-log-socket\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815769 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b013faae-082b-495b-9d77-b4371e82e379-crio-socket\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815779 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-host-run-ovn-kubernetes\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815782 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815794 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815809 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b013faae-082b-495b-9d77-b4371e82e379-crio-socket\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815831 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-run-netns\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815869 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-run-netns\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815873 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b013faae-082b-495b-9d77-b4371e82e379-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:49.815882 4493 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815908 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59643abf-923a-4d4d-8235-c2ad22c8aea0-hosts-file\") pod \"node-resolver-kzsn2\" (UID: \"59643abf-923a-4d4d-8235-c2ad22c8aea0\") " pod="openshift-dns/node-resolver-kzsn2" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:49.815924 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls podName:b013faae-082b-495b-9d77-b4371e82e379 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:50.315912869 +0000 UTC m=+1.873903345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls") pod "insights-runtime-extractor-67nxb" (UID: "b013faae-082b-495b-9d77-b4371e82e379") : secret "insights-runtime-extractor-tls" not found Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815939 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59643abf-923a-4d4d-8235-c2ad22c8aea0-hosts-file\") pod \"node-resolver-kzsn2\" (UID: \"59643abf-923a-4d4d-8235-c2ad22c8aea0\") " pod="openshift-dns/node-resolver-kzsn2" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815969 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-etc-openvswitch\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.815985 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-modprobe-d\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.816000 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-run-openvswitch\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.816030 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-etc-openvswitch\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.816038 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-etc-modprobe-d\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.816031 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-system-cni-dir\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.816003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.816038 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-run-openvswitch\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.817825 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.816064 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-system-cni-dir\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.817825 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.816068 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-run-k8s-cni-cncf-io\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.817825 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.816092 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs\") pod \"network-metrics-daemon-f28pp\" (UID: \"2281c3da-747c-4a37-89ec-b764d8a95405\") " pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:49.817825 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.816102 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5125fef-6a2d-45d6-99d2-90915b575f32-host-run-k8s-cni-cncf-io\") pod \"multus-pv8vt\" (UID: \"d5125fef-6a2d-45d6-99d2-90915b575f32\") " pod="openshift-multus/multus-pv8vt" Nov 03 20:28:49.817825 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.816121 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-sys-fs\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:49.817825 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.816136 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-host\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.817825 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.816150 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-run-systemd\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.817825 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.816165 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-registration-dir\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:49.817825 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:49.816172 4493 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 20:28:49.817825 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.816181 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-etc-selinux\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:49.817825 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.816202 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5474343-621e-4dd9-948c-ad71bd6f00c2-host\") pod \"tuned-bv4jb\" (UID: \"c5474343-621e-4dd9-948c-ad71bd6f00c2\") " pod="openshift-cluster-node-tuning-operator/tuned-bv4jb" Nov 03 20:28:49.817825 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.816204 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d48c0b65-db57-43b2-8e7c-6e9e8feddf89-run-systemd\") pod \"ovnkube-node-82vml\" (UID: \"d48c0b65-db57-43b2-8e7c-6e9e8feddf89\") " pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:49.817825 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.816219 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-sys-fs\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:49.817825 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:49.816230 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs podName:2281c3da-747c-4a37-89ec-b764d8a95405 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:50.316214605 +0000 UTC m=+1.874205100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs") pod "network-metrics-daemon-f28pp" (UID: "2281c3da-747c-4a37-89ec-b764d8a95405") : secret "metrics-daemon-secret" not found Nov 03 20:28:49.817825 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.816232 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-registration-dir\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:49.817825 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:49.816255 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3868c3d-8d3a-4c40-8821-ca116f8dcfed-etc-selinux\") pod \"aws-ebs-csi-driver-node-vkd26\" (UID: \"a3868c3d-8d3a-4c40-8821-ca116f8dcfed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkd26" Nov 03 20:28:50.320135 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:50.320105 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:50.320135 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:50.320140 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs\") pod \"network-metrics-daemon-f28pp\" (UID: \"2281c3da-747c-4a37-89ec-b764d8a95405\") " pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:50.320320 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:50.320160 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:50.320320 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:50.320244 4493 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:28:50.320320 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:50.320254 4493 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:28:50.320320 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:50.320244 4493 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 20:28:50.320320 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:50.320300 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls podName:480d6b9d-4f58-4779-a2cb-aa4b64f93881 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:51.320288762 +0000 UTC m=+2.878279239 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls") pod "dns-default-jjqxd" (UID: "480d6b9d-4f58-4779-a2cb-aa4b64f93881") : secret "dns-default-metrics-tls" not found Nov 03 20:28:50.320320 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:50.320313 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls podName:b013faae-082b-495b-9d77-b4371e82e379 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:51.320306486 +0000 UTC m=+2.878296963 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls") pod "insights-runtime-extractor-67nxb" (UID: "b013faae-082b-495b-9d77-b4371e82e379") : secret "insights-runtime-extractor-tls" not found Nov 03 20:28:50.320591 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:50.320347 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs podName:2281c3da-747c-4a37-89ec-b764d8a95405 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:51.320317744 +0000 UTC m=+2.878308225 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs") pod "network-metrics-daemon-f28pp" (UID: "2281c3da-747c-4a37-89ec-b764d8a95405") : secret "metrics-daemon-secret" not found Nov 03 20:28:51.015739 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:51.015710 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kzsn2_59643abf-923a-4d4d-8235-c2ad22c8aea0/dns-node-resolver/0.log" Nov 03 20:28:51.328098 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:51.328064 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs\") pod \"network-metrics-daemon-f28pp\" (UID: \"2281c3da-747c-4a37-89ec-b764d8a95405\") " pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:51.328265 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:51.328100 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:51.328265 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:51.328158 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:51.328265 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:51.328243 4493 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 20:28:51.328462 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:51.328276 4493 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:28:51.328462 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:51.328243 4493 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:28:51.328462 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:51.328359 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls podName:480d6b9d-4f58-4779-a2cb-aa4b64f93881 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:53.328314599 +0000 UTC m=+4.886305077 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls") pod "dns-default-jjqxd" (UID: "480d6b9d-4f58-4779-a2cb-aa4b64f93881") : secret "dns-default-metrics-tls" not found Nov 03 20:28:51.328462 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:51.328375 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs podName:2281c3da-747c-4a37-89ec-b764d8a95405 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:53.328368344 +0000 UTC m=+4.886358821 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs") pod "network-metrics-daemon-f28pp" (UID: "2281c3da-747c-4a37-89ec-b764d8a95405") : secret "metrics-daemon-secret" not found Nov 03 20:28:51.328462 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:51.328387 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls podName:b013faae-082b-495b-9d77-b4371e82e379 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:53.328380013 +0000 UTC m=+4.886370490 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls") pod "insights-runtime-extractor-67nxb" (UID: "b013faae-082b-495b-9d77-b4371e82e379") : secret "insights-runtime-extractor-tls" not found Nov 03 20:28:51.816376 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:51.816348 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5tx7t_f7ab819a-4321-4958-a5d5-230cbdbd9428/node-ca/0.log" Nov 03 20:28:52.616042 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:52.616013 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5jfs9_7e65f973-499a-4dac-af00-b485fb5dbbfe/serve-healthcheck-canary/0.log" Nov 03 20:28:53.342212 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:53.342157 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:53.342426 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:53.342247 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:53.342426 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:53.342358 4493 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:28:53.342426 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:53.342383 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs\") pod \"network-metrics-daemon-f28pp\" (UID: \"2281c3da-747c-4a37-89ec-b764d8a95405\") " pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:53.342426 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:53.342415 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls podName:b013faae-082b-495b-9d77-b4371e82e379 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:57.342402528 +0000 UTC m=+8.900393004 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls") pod "insights-runtime-extractor-67nxb" (UID: "b013faae-082b-495b-9d77-b4371e82e379") : secret "insights-runtime-extractor-tls" not found Nov 03 20:28:53.342607 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:53.342358 4493 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:28:53.342607 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:53.342460 4493 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 20:28:53.342607 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:53.342500 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls podName:480d6b9d-4f58-4779-a2cb-aa4b64f93881 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:57.342483545 +0000 UTC m=+8.900474045 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls") pod "dns-default-jjqxd" (UID: "480d6b9d-4f58-4779-a2cb-aa4b64f93881") : secret "dns-default-metrics-tls" not found Nov 03 20:28:53.342607 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:53.342515 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs podName:2281c3da-747c-4a37-89ec-b764d8a95405 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:57.342507651 +0000 UTC m=+8.900498127 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs") pod "network-metrics-daemon-f28pp" (UID: "2281c3da-747c-4a37-89ec-b764d8a95405") : secret "metrics-daemon-secret" not found Nov 03 20:28:53.416693 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:53.416657 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-gj9nc_3514724f-deea-4b4e-bce9-48ce7862d6f0/migrator/0.log" Nov 03 20:28:53.968014 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:53.967984 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ccxwr" Nov 03 20:28:55.086493 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:55.086453 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:28:57.361583 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:57.361544 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs\") pod \"network-metrics-daemon-f28pp\" (UID: \"2281c3da-747c-4a37-89ec-b764d8a95405\") " pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:28:57.361583 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:57.361586 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:28:57.362164 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:57.361628 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:28:57.362164 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:57.361693 4493 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 20:28:57.362164 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:57.361710 4493 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:28:57.362164 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:57.361760 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls podName:b013faae-082b-495b-9d77-b4371e82e379 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:05.361747662 +0000 UTC m=+16.919738140 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls") pod "insights-runtime-extractor-67nxb" (UID: "b013faae-082b-495b-9d77-b4371e82e379") : secret "insights-runtime-extractor-tls" not found Nov 03 20:28:57.362164 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:57.361772 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs podName:2281c3da-747c-4a37-89ec-b764d8a95405 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:05.361766301 +0000 UTC m=+16.919756777 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs") pod "network-metrics-daemon-f28pp" (UID: "2281c3da-747c-4a37-89ec-b764d8a95405") : secret "metrics-daemon-secret" not found Nov 03 20:28:57.362164 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:57.361776 4493 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:28:57.362164 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:28:57.361873 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls podName:480d6b9d-4f58-4779-a2cb-aa4b64f93881 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:05.361855579 +0000 UTC m=+16.919846062 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls") pod "dns-default-jjqxd" (UID: "480d6b9d-4f58-4779-a2cb-aa4b64f93881") : secret "dns-default-metrics-tls" not found Nov 03 20:28:58.034753 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:58.034725 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:58.703612 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:58.703576 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:28:58.704017 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:58.703757 4493 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:28:59.089535 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:59.089509 4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-26.us-west-2.compute.internal" event="NodeReady" Nov 03 20:28:59.739952 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:28:59.739927 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-82vml" Nov 03 20:29:05.413559 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:05.413507 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs\") pod \"network-metrics-daemon-f28pp\" (UID: \"2281c3da-747c-4a37-89ec-b764d8a95405\") " pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:29:05.413559 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:05.413569 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:29:05.414075 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:05.413626 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:29:05.414075 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:29:05.413650 4493 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 20:29:05.414075 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:29:05.413724 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs podName:2281c3da-747c-4a37-89ec-b764d8a95405 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:21.413709861 +0000 UTC m=+32.971700351 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs") pod "network-metrics-daemon-f28pp" (UID: "2281c3da-747c-4a37-89ec-b764d8a95405") : secret "metrics-daemon-secret" not found Nov 03 20:29:05.414075 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:29:05.413738 4493 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:29:05.414075 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:29:05.413738 4493 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:29:05.414075 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:29:05.413804 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls podName:480d6b9d-4f58-4779-a2cb-aa4b64f93881 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:21.413788212 +0000 UTC m=+32.971778705 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls") pod "dns-default-jjqxd" (UID: "480d6b9d-4f58-4779-a2cb-aa4b64f93881") : secret "dns-default-metrics-tls" not found Nov 03 20:29:05.414075 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:29:05.413820 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls podName:b013faae-082b-495b-9d77-b4371e82e379 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:21.41381341 +0000 UTC m=+32.971803887 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls") pod "insights-runtime-extractor-67nxb" (UID: "b013faae-082b-495b-9d77-b4371e82e379") : secret "insights-runtime-extractor-tls" not found Nov 03 20:29:09.542816 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.542777 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65c4b8fd96-jpckm"] Nov 03 20:29:09.547171 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.547148 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:09.620514 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.620492 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 20:29:09.639314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.639284 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2296e3c6-6709-4fb0-8384-b0800d874d32-registry-certificates\") pod \"image-registry-65c4b8fd96-jpckm\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:09.639452 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.639317 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsfrj\" (UniqueName: \"kubernetes.io/projected/2296e3c6-6709-4fb0-8384-b0800d874d32-kube-api-access-vsfrj\") pod \"image-registry-65c4b8fd96-jpckm\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:09.639452 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.639368 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2296e3c6-6709-4fb0-8384-b0800d874d32-bound-sa-token\") pod \"image-registry-65c4b8fd96-jpckm\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:09.639452 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.639396 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2296e3c6-6709-4fb0-8384-b0800d874d32-trusted-ca\") pod \"image-registry-65c4b8fd96-jpckm\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:09.639452 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.639420 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2296e3c6-6709-4fb0-8384-b0800d874d32-installation-pull-secrets\") pod \"image-registry-65c4b8fd96-jpckm\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:09.639636 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.639506 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2296e3c6-6709-4fb0-8384-b0800d874d32-image-registry-private-configuration\") pod \"image-registry-65c4b8fd96-jpckm\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:09.639636 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.639555 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2296e3c6-6709-4fb0-8384-b0800d874d32-registry-tls\") pod \"image-registry-65c4b8fd96-jpckm\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:09.639636 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.639578 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2296e3c6-6709-4fb0-8384-b0800d874d32-ca-trust-extracted\") pod \"image-registry-65c4b8fd96-jpckm\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:09.643723 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.643703 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 20:29:09.643847 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.643740 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mwh5m\"" Nov 03 20:29:09.643847 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.643816 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 20:29:09.738275 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.738244 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 20:29:09.738881 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.738854 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65c4b8fd96-jpckm"] Nov 03 20:29:09.740799 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.740775 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2296e3c6-6709-4fb0-8384-b0800d874d32-registry-tls\") pod \"image-registry-65c4b8fd96-jpckm\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:09.740877 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.740820 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2296e3c6-6709-4fb0-8384-b0800d874d32-ca-trust-extracted\") pod \"image-registry-65c4b8fd96-jpckm\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:09.740933 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.740890 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2296e3c6-6709-4fb0-8384-b0800d874d32-registry-certificates\") pod \"image-registry-65c4b8fd96-jpckm\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:09.740933 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.740913 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsfrj\" (UniqueName: \"kubernetes.io/projected/2296e3c6-6709-4fb0-8384-b0800d874d32-kube-api-access-vsfrj\") pod \"image-registry-65c4b8fd96-jpckm\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:09.741038 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.741018 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2296e3c6-6709-4fb0-8384-b0800d874d32-bound-sa-token\") pod \"image-registry-65c4b8fd96-jpckm\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:09.741090 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.741062 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2296e3c6-6709-4fb0-8384-b0800d874d32-trusted-ca\") pod \"image-registry-65c4b8fd96-jpckm\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:09.741139 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.741089 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2296e3c6-6709-4fb0-8384-b0800d874d32-installation-pull-secrets\") pod \"image-registry-65c4b8fd96-jpckm\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:09.741190 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.741135 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2296e3c6-6709-4fb0-8384-b0800d874d32-image-registry-private-configuration\") pod \"image-registry-65c4b8fd96-jpckm\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:09.741241 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.741201 4493 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 20:29:09.741475 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.741456 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2296e3c6-6709-4fb0-8384-b0800d874d32-ca-trust-extracted\") pod \"image-registry-65c4b8fd96-jpckm\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:09.741787 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.741765 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2296e3c6-6709-4fb0-8384-b0800d874d32-registry-certificates\") pod \"image-registry-65c4b8fd96-jpckm\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:09.742164 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.742142 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2296e3c6-6709-4fb0-8384-b0800d874d32-trusted-ca\") pod \"image-registry-65c4b8fd96-jpckm\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:09.743649 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.743628 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2296e3c6-6709-4fb0-8384-b0800d874d32-image-registry-private-configuration\") pod \"image-registry-65c4b8fd96-jpckm\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:09.744065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.744043 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2296e3c6-6709-4fb0-8384-b0800d874d32-installation-pull-secrets\") pod \"image-registry-65c4b8fd96-jpckm\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:09.744152 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.744137 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2296e3c6-6709-4fb0-8384-b0800d874d32-registry-tls\") pod \"image-registry-65c4b8fd96-jpckm\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:09.794673 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.794588 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2296e3c6-6709-4fb0-8384-b0800d874d32-bound-sa-token\") pod \"image-registry-65c4b8fd96-jpckm\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:09.794673 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.794613 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsfrj\" (UniqueName: \"kubernetes.io/projected/2296e3c6-6709-4fb0-8384-b0800d874d32-kube-api-access-vsfrj\") pod \"image-registry-65c4b8fd96-jpckm\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:09.857558 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:09.857523 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:10.326667 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:10.326634 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65c4b8fd96-jpckm"] Nov 03 20:29:10.371803 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:29:10.371776 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2296e3c6_6709_4fb0_8384_b0800d874d32.slice/crio-7b707c0e0bb949bf2afc47ea2907950196440e8b2fd16d0a74a14cf83d74b9a2 WatchSource:0}: Error finding container 7b707c0e0bb949bf2afc47ea2907950196440e8b2fd16d0a74a14cf83d74b9a2: Status 404 returned error can't find the container with id 7b707c0e0bb949bf2afc47ea2907950196440e8b2fd16d0a74a14cf83d74b9a2 Nov 03 20:29:10.644656 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:10.644575 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" event={"ID":"2296e3c6-6709-4fb0-8384-b0800d874d32","Type":"ContainerStarted","Data":"1dcfed70aa4685cb24bd7042f86afcb11facd2af48147910c1168142d34f899f"} Nov 03 20:29:10.644656 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:10.644630 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" event={"ID":"2296e3c6-6709-4fb0-8384-b0800d874d32","Type":"ContainerStarted","Data":"7b707c0e0bb949bf2afc47ea2907950196440e8b2fd16d0a74a14cf83d74b9a2"} Nov 03 20:29:10.645049 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:10.644671 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:10.757232 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:10.757152 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" podStartSLOduration=1.7571378050000002 podStartE2EDuration="1.757137805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:10.75712567 +0000 UTC m=+22.315116188" watchObservedRunningTime="2026-11-03 20:29:10.757137805 +0000 UTC m=+22.315128303" Nov 03 20:29:21.416714 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:21.416679 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:29:21.417241 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:21.416724 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs\") pod \"network-metrics-daemon-f28pp\" (UID: \"2281c3da-747c-4a37-89ec-b764d8a95405\") " pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:29:21.417241 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:21.416863 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:29:21.419265 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:21.419233 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b013faae-082b-495b-9d77-b4371e82e379-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67nxb\" (UID: \"b013faae-082b-495b-9d77-b4371e82e379\") " pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:29:21.419265 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:21.419248 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2281c3da-747c-4a37-89ec-b764d8a95405-metrics-certs\") pod \"network-metrics-daemon-f28pp\" (UID: \"2281c3da-747c-4a37-89ec-b764d8a95405\") " pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:29:21.419437 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:21.419248 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/480d6b9d-4f58-4779-a2cb-aa4b64f93881-metrics-tls\") pod \"dns-default-jjqxd\" (UID: \"480d6b9d-4f58-4779-a2cb-aa4b64f93881\") " pod="openshift-dns/dns-default-jjqxd" Nov 03 20:29:21.714302 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:21.714228 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f28pp" Nov 03 20:29:21.715349 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:21.715313 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jjqxd" Nov 03 20:29:21.716418 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:21.716401 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-67nxb" Nov 03 20:29:22.121822 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:22.121784 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f28pp"] Nov 03 20:29:22.170954 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:29:22.170926 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2281c3da_747c_4a37_89ec_b764d8a95405.slice/crio-c5ca8b57dfc294a0f4f64aa339e1523e94e7c546b77ff4625d64813f874848b1 WatchSource:0}: Error finding container c5ca8b57dfc294a0f4f64aa339e1523e94e7c546b77ff4625d64813f874848b1: Status 404 returned error can't find the container with id c5ca8b57dfc294a0f4f64aa339e1523e94e7c546b77ff4625d64813f874848b1 Nov 03 20:29:22.172933 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:22.172912 4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:29:22.174037 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:22.174017 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jjqxd"] Nov 03 20:29:22.180679 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:29:22.180654 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod480d6b9d_4f58_4779_a2cb_aa4b64f93881.slice/crio-fb2b3ce481627988f28e15132bd8bc0b91230f2da709e277d93b9037b765a70a WatchSource:0}: Error finding container fb2b3ce481627988f28e15132bd8bc0b91230f2da709e277d93b9037b765a70a: Status 404 returned error can't find the container with id fb2b3ce481627988f28e15132bd8bc0b91230f2da709e277d93b9037b765a70a Nov 03 20:29:22.220926 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:22.220907 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-67nxb"] Nov 03 20:29:22.222187 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:29:22.222165 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb013faae_082b_495b_9d77_b4371e82e379.slice/crio-29b61969e68df2127984de76cbb725afa8916acc3d4665b06f7883d7dade1f63 WatchSource:0}: Error finding container 29b61969e68df2127984de76cbb725afa8916acc3d4665b06f7883d7dade1f63: Status 404 returned error can't find the container with id 29b61969e68df2127984de76cbb725afa8916acc3d4665b06f7883d7dade1f63 Nov 03 20:29:22.673111 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:22.673079 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-67nxb" event={"ID":"b013faae-082b-495b-9d77-b4371e82e379","Type":"ContainerStarted","Data":"35b2ec4386fb9ac37fdb0acf0632c5d06ec75bfc3fda6dc849bcac20a97af551"} Nov 03 20:29:22.673527 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:22.673136 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-67nxb" event={"ID":"b013faae-082b-495b-9d77-b4371e82e379","Type":"ContainerStarted","Data":"29b61969e68df2127984de76cbb725afa8916acc3d4665b06f7883d7dade1f63"} Nov 03 20:29:22.674302 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:22.674277 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jjqxd" event={"ID":"480d6b9d-4f58-4779-a2cb-aa4b64f93881","Type":"ContainerStarted","Data":"fb2b3ce481627988f28e15132bd8bc0b91230f2da709e277d93b9037b765a70a"} Nov 03 20:29:22.675423 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:22.675402 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f28pp" event={"ID":"2281c3da-747c-4a37-89ec-b764d8a95405","Type":"ContainerStarted","Data":"c5ca8b57dfc294a0f4f64aa339e1523e94e7c546b77ff4625d64813f874848b1"} Nov 03 20:29:23.524402 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:23.524364 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-wrlml-k5rcz"] Nov 03 20:29:23.527385 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:23.527367 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wrlml-k5rcz" Nov 03 20:29:23.580992 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:23.580958 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wrlml-k5rcz"] Nov 03 20:29:23.637900 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:23.637869 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-service-ca-bundle\") pod \"periodic-gathering-wrlml-k5rcz\" (UID: \"08c9d7c8-dbd8-46ad-92f4-2a009e83abc3\") " pod="openshift-insights/periodic-gathering-wrlml-k5rcz" Nov 03 20:29:23.637900 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:23.637902 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqrdc\" (UniqueName: \"kubernetes.io/projected/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-kube-api-access-xqrdc\") pod \"periodic-gathering-wrlml-k5rcz\" (UID: \"08c9d7c8-dbd8-46ad-92f4-2a009e83abc3\") " pod="openshift-insights/periodic-gathering-wrlml-k5rcz" Nov 03 20:29:23.638081 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:23.637998 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-archives-path\") pod \"periodic-gathering-wrlml-k5rcz\" (UID: \"08c9d7c8-dbd8-46ad-92f4-2a009e83abc3\") " pod="openshift-insights/periodic-gathering-wrlml-k5rcz" Nov 03 20:29:23.638081 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:23.638067 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-trusted-ca-bundle\") pod \"periodic-gathering-wrlml-k5rcz\" (UID: \"08c9d7c8-dbd8-46ad-92f4-2a009e83abc3\") " pod="openshift-insights/periodic-gathering-wrlml-k5rcz" Nov 03 20:29:23.681292 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:23.681253 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-67nxb" event={"ID":"b013faae-082b-495b-9d77-b4371e82e379","Type":"ContainerStarted","Data":"9c53c03907c00c30cc4f975d0bf281353a846d36a699142b1df059ffca32119d"} Nov 03 20:29:23.738635 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:23.738607 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-service-ca-bundle\") pod \"periodic-gathering-wrlml-k5rcz\" (UID: \"08c9d7c8-dbd8-46ad-92f4-2a009e83abc3\") " pod="openshift-insights/periodic-gathering-wrlml-k5rcz" Nov 03 20:29:23.738784 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:23.738688 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqrdc\" (UniqueName: \"kubernetes.io/projected/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-kube-api-access-xqrdc\") pod \"periodic-gathering-wrlml-k5rcz\" (UID: \"08c9d7c8-dbd8-46ad-92f4-2a009e83abc3\") " pod="openshift-insights/periodic-gathering-wrlml-k5rcz" Nov 03 20:29:23.738851 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:23.738787 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-archives-path\") pod \"periodic-gathering-wrlml-k5rcz\" (UID: \"08c9d7c8-dbd8-46ad-92f4-2a009e83abc3\") " pod="openshift-insights/periodic-gathering-wrlml-k5rcz" Nov 03 20:29:23.738851 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:23.738816 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-trusted-ca-bundle\") pod \"periodic-gathering-wrlml-k5rcz\" (UID: \"08c9d7c8-dbd8-46ad-92f4-2a009e83abc3\") " pod="openshift-insights/periodic-gathering-wrlml-k5rcz" Nov 03 20:29:23.739181 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:23.739150 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-archives-path\") pod \"periodic-gathering-wrlml-k5rcz\" (UID: \"08c9d7c8-dbd8-46ad-92f4-2a009e83abc3\") " pod="openshift-insights/periodic-gathering-wrlml-k5rcz" Nov 03 20:29:23.739395 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:23.739378 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-service-ca-bundle\") pod \"periodic-gathering-wrlml-k5rcz\" (UID: \"08c9d7c8-dbd8-46ad-92f4-2a009e83abc3\") " pod="openshift-insights/periodic-gathering-wrlml-k5rcz" Nov 03 20:29:23.739646 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:23.739628 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-trusted-ca-bundle\") pod \"periodic-gathering-wrlml-k5rcz\" (UID: \"08c9d7c8-dbd8-46ad-92f4-2a009e83abc3\") " pod="openshift-insights/periodic-gathering-wrlml-k5rcz" Nov 03 20:29:23.792801 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:23.792748 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqrdc\" (UniqueName: \"kubernetes.io/projected/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-kube-api-access-xqrdc\") pod \"periodic-gathering-wrlml-k5rcz\" (UID: \"08c9d7c8-dbd8-46ad-92f4-2a009e83abc3\") " pod="openshift-insights/periodic-gathering-wrlml-k5rcz" Nov 03 20:29:23.843707 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:23.843677 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wrlml-k5rcz" Nov 03 20:29:24.210397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:24.210359 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wrlml-k5rcz"] Nov 03 20:29:24.219431 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:29:24.219400 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c9d7c8_dbd8_46ad_92f4_2a009e83abc3.slice/crio-4fa25eaa7a3420c901cd2058dccc33d4869a9255668c691a5fa2b6f4a8d57e4c WatchSource:0}: Error finding container 4fa25eaa7a3420c901cd2058dccc33d4869a9255668c691a5fa2b6f4a8d57e4c: Status 404 returned error can't find the container with id 4fa25eaa7a3420c901cd2058dccc33d4869a9255668c691a5fa2b6f4a8d57e4c Nov 03 20:29:24.689479 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:24.689439 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wrlml-k5rcz" event={"ID":"08c9d7c8-dbd8-46ad-92f4-2a009e83abc3","Type":"ContainerStarted","Data":"a054fcf5f233995135e99c64f3fbff1dec8e78e45b1033764e47cc27ba1f528e"} Nov 03 20:29:24.689895 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:24.689505 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wrlml-k5rcz" event={"ID":"08c9d7c8-dbd8-46ad-92f4-2a009e83abc3","Type":"ContainerStarted","Data":"4fa25eaa7a3420c901cd2058dccc33d4869a9255668c691a5fa2b6f4a8d57e4c"} Nov 03 20:29:24.797209 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:24.797094 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-wrlml-k5rcz" podStartSLOduration=1.797062677 podStartE2EDuration="1.797062677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:24.795850118 +0000 UTC m=+36.353840643" watchObservedRunningTime="2026-11-03 20:29:24.797062677 +0000 UTC m=+36.355053167" Nov 03 20:29:25.089292 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:25.089262 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-l4pbv" Nov 03 20:29:25.695115 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:25.695072 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-67nxb" event={"ID":"b013faae-082b-495b-9d77-b4371e82e379","Type":"ContainerStarted","Data":"5fe925bbc4fd1fba2295e6cec8008381b3e0fff894702b1f0563b6d2ce0925bb"} Nov 03 20:29:25.696915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:25.696886 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f28pp" event={"ID":"2281c3da-747c-4a37-89ec-b764d8a95405","Type":"ContainerStarted","Data":"b3bd13ce58150875256465e31f4bef561bd210ea76a2989a95662e41fa5721b6"} Nov 03 20:29:25.697025 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:25.696938 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f28pp" event={"ID":"2281c3da-747c-4a37-89ec-b764d8a95405","Type":"ContainerStarted","Data":"3b2336802d19319b4af8b66ec34f5abb8e50063c3c86b2f36f73a2e560b41af7"} Nov 03 20:29:27.219996 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:27.219965 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kzsn2_59643abf-923a-4d4d-8235-c2ad22c8aea0/dns-node-resolver/0.log" Nov 03 20:29:27.821610 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:27.821578 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-65c4b8fd96-jpckm_2296e3c6-6709-4fb0-8384-b0800d874d32/registry/0.log" Nov 03 20:29:28.019621 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:28.019596 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5tx7t_f7ab819a-4321-4958-a5d5-230cbdbd9428/node-ca/0.log" Nov 03 20:29:28.820836 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:28.820755 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5jfs9_7e65f973-499a-4dac-af00-b485fb5dbbfe/serve-healthcheck-canary/0.log" Nov 03 20:29:29.719019 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:29.718982 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jjqxd" event={"ID":"480d6b9d-4f58-4779-a2cb-aa4b64f93881","Type":"ContainerStarted","Data":"4032c2fce30b9f5e9655d874d526c7f9bd389086159826ac1fe85b0e11968981"} Nov 03 20:29:29.719212 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:29.719032 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jjqxd" event={"ID":"480d6b9d-4f58-4779-a2cb-aa4b64f93881","Type":"ContainerStarted","Data":"8ca0ff6fe48dfbb8ddd73e0d01a0f976122184ada31b8eaa535730c39f08d0f0"} Nov 03 20:29:29.719212 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:29.719064 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jjqxd" Nov 03 20:29:29.861539 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:29.861507 4493 patch_prober.go:29] interesting pod/image-registry-65c4b8fd96-jpckm container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 20:29:29.861913 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:29.861559 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" podUID="2296e3c6-6709-4fb0-8384-b0800d874d32" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:29:31.651005 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:31.650973 4493 patch_prober.go:29] interesting pod/image-registry-65c4b8fd96-jpckm container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 20:29:31.651398 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:31.651026 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" podUID="2296e3c6-6709-4fb0-8384-b0800d874d32" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:29:33.546895 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:33.546861 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-bbqpt"] Nov 03 20:29:33.549366 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:33.549350 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bbqpt" Nov 03 20:29:33.609545 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:33.609512 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4pvg\" (UniqueName: \"kubernetes.io/projected/1fb24a85-7171-4924-9578-fe5e5fe58ca5-kube-api-access-b4pvg\") pod \"prometheus-operator-9cd89b74c-bbqpt\" (UID: \"1fb24a85-7171-4924-9578-fe5e5fe58ca5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bbqpt" Nov 03 20:29:33.609706 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:33.609638 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fb24a85-7171-4924-9578-fe5e5fe58ca5-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-bbqpt\" (UID: \"1fb24a85-7171-4924-9578-fe5e5fe58ca5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bbqpt" Nov 03 20:29:33.609706 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:33.609677 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1fb24a85-7171-4924-9578-fe5e5fe58ca5-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-bbqpt\" (UID: \"1fb24a85-7171-4924-9578-fe5e5fe58ca5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bbqpt" Nov 03 20:29:33.609833 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:33.609746 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1fb24a85-7171-4924-9578-fe5e5fe58ca5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-bbqpt\" (UID: \"1fb24a85-7171-4924-9578-fe5e5fe58ca5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bbqpt" Nov 03 20:29:33.645891 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:33.645870 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 20:29:33.646056 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:33.646040 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 20:29:33.646114 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:33.646061 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 20:29:33.646114 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:33.646105 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 20:29:33.646197 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:33.646110 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 20:29:33.646197 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:33.646139 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vh75h\"" Nov 03 20:29:33.693685 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:33.693661 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-bbqpt"] Nov 03 20:29:33.710521 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:33.710500 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fb24a85-7171-4924-9578-fe5e5fe58ca5-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-bbqpt\" (UID: \"1fb24a85-7171-4924-9578-fe5e5fe58ca5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bbqpt" Nov 03 20:29:33.710625 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:33.710529 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1fb24a85-7171-4924-9578-fe5e5fe58ca5-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-bbqpt\" (UID: \"1fb24a85-7171-4924-9578-fe5e5fe58ca5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bbqpt" Nov 03 20:29:33.710625 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:33.710552 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1fb24a85-7171-4924-9578-fe5e5fe58ca5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-bbqpt\" (UID: \"1fb24a85-7171-4924-9578-fe5e5fe58ca5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bbqpt" Nov 03 20:29:33.710625 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:33.710584 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4pvg\" (UniqueName: \"kubernetes.io/projected/1fb24a85-7171-4924-9578-fe5e5fe58ca5-kube-api-access-b4pvg\") pod \"prometheus-operator-9cd89b74c-bbqpt\" (UID: \"1fb24a85-7171-4924-9578-fe5e5fe58ca5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bbqpt" Nov 03 20:29:33.711183 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:33.711164 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fb24a85-7171-4924-9578-fe5e5fe58ca5-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-bbqpt\" (UID: \"1fb24a85-7171-4924-9578-fe5e5fe58ca5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bbqpt" Nov 03 20:29:33.713109 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:33.713091 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1fb24a85-7171-4924-9578-fe5e5fe58ca5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-bbqpt\" (UID: \"1fb24a85-7171-4924-9578-fe5e5fe58ca5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bbqpt" Nov 03 20:29:33.713210 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:33.713166 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1fb24a85-7171-4924-9578-fe5e5fe58ca5-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-bbqpt\" (UID: \"1fb24a85-7171-4924-9578-fe5e5fe58ca5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bbqpt" Nov 03 20:29:33.763992 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:33.763963 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4pvg\" (UniqueName: \"kubernetes.io/projected/1fb24a85-7171-4924-9578-fe5e5fe58ca5-kube-api-access-b4pvg\") pod \"prometheus-operator-9cd89b74c-bbqpt\" (UID: \"1fb24a85-7171-4924-9578-fe5e5fe58ca5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bbqpt" Nov 03 20:29:33.860184 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:33.860120 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bbqpt" Nov 03 20:29:34.222739 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:34.222668 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-bbqpt"] Nov 03 20:29:34.261901 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:29:34.261876 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb24a85_7171_4924_9578_fe5e5fe58ca5.slice/crio-bdb6ce93960ec9aa21140188e19c9195903b885a7b13dd4469d9f12768100bac WatchSource:0}: Error finding container bdb6ce93960ec9aa21140188e19c9195903b885a7b13dd4469d9f12768100bac: Status 404 returned error can't find the container with id bdb6ce93960ec9aa21140188e19c9195903b885a7b13dd4469d9f12768100bac Nov 03 20:29:34.736087 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:34.736052 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bbqpt" event={"ID":"1fb24a85-7171-4924-9578-fe5e5fe58ca5","Type":"ContainerStarted","Data":"bdb6ce93960ec9aa21140188e19c9195903b885a7b13dd4469d9f12768100bac"} Nov 03 20:29:39.724903 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:39.724877 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jjqxd" Nov 03 20:29:39.866495 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:39.862899 4493 patch_prober.go:29] interesting pod/image-registry-65c4b8fd96-jpckm container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 20:29:39.866495 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:39.863922 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" podUID="2296e3c6-6709-4fb0-8384-b0800d874d32" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:29:40.648326 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:40.648289 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-rr8h2-bjl4n"] Nov 03 20:29:40.648887 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:40.648669 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" podUID="34b6b405-a677-4fef-8ee6-37d84a1c5902" containerName="insights-gathering" containerID="cri-o://4dca622fd78277fac0622261e66b08c483bbc760b969cd1b39386d5165d46c20" gracePeriod=30 Nov 03 20:29:40.897585 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:40.897561 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" Nov 03 20:29:40.959052 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:40.958990 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b6b405-a677-4fef-8ee6-37d84a1c5902-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34b6b405-a677-4fef-8ee6-37d84a1c5902-service-ca-bundle\") pod \"34b6b405-a677-4fef-8ee6-37d84a1c5902\" (UID: \"34b6b405-a677-4fef-8ee6-37d84a1c5902\") " Nov 03 20:29:40.959052 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:40.959029 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b6b405-a677-4fef-8ee6-37d84a1c5902-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/34b6b405-a677-4fef-8ee6-37d84a1c5902-archives-path\") pod \"34b6b405-a677-4fef-8ee6-37d84a1c5902\" (UID: \"34b6b405-a677-4fef-8ee6-37d84a1c5902\") " Nov 03 20:29:40.959052 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:40.959052 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b6b405-a677-4fef-8ee6-37d84a1c5902-kube-api-access-h9qrh\" (UniqueName: \"kubernetes.io/projected/34b6b405-a677-4fef-8ee6-37d84a1c5902-kube-api-access-h9qrh\") pod \"34b6b405-a677-4fef-8ee6-37d84a1c5902\" (UID: \"34b6b405-a677-4fef-8ee6-37d84a1c5902\") " Nov 03 20:29:40.959256 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:40.959108 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b6b405-a677-4fef-8ee6-37d84a1c5902-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34b6b405-a677-4fef-8ee6-37d84a1c5902-trusted-ca-bundle\") pod \"34b6b405-a677-4fef-8ee6-37d84a1c5902\" (UID: \"34b6b405-a677-4fef-8ee6-37d84a1c5902\") " Nov 03 20:29:40.959462 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:40.959434 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b6b405-a677-4fef-8ee6-37d84a1c5902-archives-path" pod "34b6b405-a677-4fef-8ee6-37d84a1c5902" (UID: "34b6b405-a677-4fef-8ee6-37d84a1c5902"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:40.959548 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:40.959306 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b6b405-a677-4fef-8ee6-37d84a1c5902-service-ca-bundle" pod "34b6b405-a677-4fef-8ee6-37d84a1c5902" (UID: "34b6b405-a677-4fef-8ee6-37d84a1c5902"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:40.959586 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:40.959539 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b6b405-a677-4fef-8ee6-37d84a1c5902-trusted-ca-bundle" pod "34b6b405-a677-4fef-8ee6-37d84a1c5902" (UID: "34b6b405-a677-4fef-8ee6-37d84a1c5902"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:40.961541 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:40.961517 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b6b405-a677-4fef-8ee6-37d84a1c5902-kube-api-access-h9qrh" pod "34b6b405-a677-4fef-8ee6-37d84a1c5902" (UID: "34b6b405-a677-4fef-8ee6-37d84a1c5902"). InnerVolumeSpecName "kube-api-access-h9qrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:41.059905 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:41.059881 4493 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34b6b405-a677-4fef-8ee6-37d84a1c5902-service-ca-bundle\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:41.059905 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:41.059907 4493 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/34b6b405-a677-4fef-8ee6-37d84a1c5902-archives-path\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:41.060041 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:41.059917 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9qrh\" (UniqueName: \"kubernetes.io/projected/34b6b405-a677-4fef-8ee6-37d84a1c5902-kube-api-access-h9qrh\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:41.060041 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:41.059927 4493 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34b6b405-a677-4fef-8ee6-37d84a1c5902-trusted-ca-bundle\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:41.651673 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:41.651636 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:29:41.759799 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:41.759764 4493 generic.go:356] "Generic (PLEG): container finished" podID="34b6b405-a677-4fef-8ee6-37d84a1c5902" containerID="4dca622fd78277fac0622261e66b08c483bbc760b969cd1b39386d5165d46c20" exitCode=2 Nov 03 20:29:41.759987 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:41.759801 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" event={"ID":"34b6b405-a677-4fef-8ee6-37d84a1c5902","Type":"ContainerDied","Data":"4dca622fd78277fac0622261e66b08c483bbc760b969cd1b39386d5165d46c20"} Nov 03 20:29:41.759987 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:41.759853 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" event={"ID":"34b6b405-a677-4fef-8ee6-37d84a1c5902","Type":"ContainerDied","Data":"14f0e7044649ef39b41f8df4f11301f5b107fb03376661ebd8b18605a9953596"} Nov 03 20:29:41.759987 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:41.759811 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rr8h2-bjl4n" Nov 03 20:29:41.759987 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:41.759873 4493 scope.go:122] "RemoveContainer" containerID="4dca622fd78277fac0622261e66b08c483bbc760b969cd1b39386d5165d46c20" Nov 03 20:29:41.761861 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:41.761833 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bbqpt" event={"ID":"1fb24a85-7171-4924-9578-fe5e5fe58ca5","Type":"ContainerStarted","Data":"5d3f087529beccf378546ffafecbda80016c92d886b870b455f8b61c45ad165f"} Nov 03 20:29:41.762006 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:41.761882 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bbqpt" event={"ID":"1fb24a85-7171-4924-9578-fe5e5fe58ca5","Type":"ContainerStarted","Data":"b2cb365244b780a366c28484cd994245d686f0cf0d81dbce5ed8a575e3e21a56"} Nov 03 20:29:41.768898 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:41.768878 4493 scope.go:122] "RemoveContainer" containerID="4dca622fd78277fac0622261e66b08c483bbc760b969cd1b39386d5165d46c20" Nov 03 20:29:41.769139 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:29:41.769117 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dca622fd78277fac0622261e66b08c483bbc760b969cd1b39386d5165d46c20\": container with ID starting with 4dca622fd78277fac0622261e66b08c483bbc760b969cd1b39386d5165d46c20 not found: ID does not exist" containerID="4dca622fd78277fac0622261e66b08c483bbc760b969cd1b39386d5165d46c20" Nov 03 20:29:41.769222 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:41.769143 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dca622fd78277fac0622261e66b08c483bbc760b969cd1b39386d5165d46c20"} err="failed to get container status \"4dca622fd78277fac0622261e66b08c483bbc760b969cd1b39386d5165d46c20\": rpc error: code = NotFound desc = could not find container \"4dca622fd78277fac0622261e66b08c483bbc760b969cd1b39386d5165d46c20\": container with ID starting with 4dca622fd78277fac0622261e66b08c483bbc760b969cd1b39386d5165d46c20 not found: ID does not exist" Nov 03 20:29:41.870479 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:41.870434 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bbqpt" podStartSLOduration=2.143832284 podStartE2EDuration="8.870420524s" totalImagesPullingTime="6.72658824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:41.869455463 +0000 UTC m=+53.427445981" watchObservedRunningTime="2026-11-03 20:29:41.870420524 +0000 UTC m=+53.428411023" Nov 03 20:29:42.022306 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:42.022236 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-rr8h2-bjl4n"] Nov 03 20:29:42.024078 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:42.024055 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-rr8h2-bjl4n"] Nov 03 20:29:42.559085 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:42.559053 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b6b405-a677-4fef-8ee6-37d84a1c5902" path="/var/lib/kubelet/pods/34b6b405-a677-4fef-8ee6-37d84a1c5902/volumes" Nov 03 20:29:44.178727 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:44.178694 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65c4b8fd96-jpckm"] Nov 03 20:29:45.002922 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.002891 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-5ckdt"] Nov 03 20:29:45.003143 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.003130 4493 cpu_manager.go:414] "containerMap: removing container" podUID="34b6b405-a677-4fef-8ee6-37d84a1c5902" containerName="insights-gathering" Nov 03 20:29:45.003187 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.003144 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b6b405-a677-4fef-8ee6-37d84a1c5902" containerName="insights-gathering" Nov 03 20:29:45.003224 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.003195 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b6b405-a677-4fef-8ee6-37d84a1c5902" containerName="insights-gathering" Nov 03 20:29:45.006497 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.006476 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5ckdt" Nov 03 20:29:45.076621 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.076594 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 20:29:45.078116 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.078103 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 20:29:45.078544 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.078527 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-76lq4\"" Nov 03 20:29:45.085871 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.085841 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/09f023f0-38ef-4d16-a5d7-f3e309d853ba-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5ckdt\" (UID: \"09f023f0-38ef-4d16-a5d7-f3e309d853ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5ckdt" Nov 03 20:29:45.085975 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.085888 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09f023f0-38ef-4d16-a5d7-f3e309d853ba-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5ckdt\" (UID: \"09f023f0-38ef-4d16-a5d7-f3e309d853ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5ckdt" Nov 03 20:29:45.086027 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.085992 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09f023f0-38ef-4d16-a5d7-f3e309d853ba-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5ckdt\" (UID: \"09f023f0-38ef-4d16-a5d7-f3e309d853ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5ckdt" Nov 03 20:29:45.086027 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.086019 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g9km\" (UniqueName: \"kubernetes.io/projected/09f023f0-38ef-4d16-a5d7-f3e309d853ba-kube-api-access-5g9km\") pod \"openshift-state-metrics-7bd7489957-5ckdt\" (UID: \"09f023f0-38ef-4d16-a5d7-f3e309d853ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5ckdt" Nov 03 20:29:45.122116 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.122087 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-5ckdt"] Nov 03 20:29:45.186617 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.186583 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09f023f0-38ef-4d16-a5d7-f3e309d853ba-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5ckdt\" (UID: \"09f023f0-38ef-4d16-a5d7-f3e309d853ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5ckdt" Nov 03 20:29:45.186617 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.186622 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g9km\" (UniqueName: \"kubernetes.io/projected/09f023f0-38ef-4d16-a5d7-f3e309d853ba-kube-api-access-5g9km\") pod \"openshift-state-metrics-7bd7489957-5ckdt\" (UID: \"09f023f0-38ef-4d16-a5d7-f3e309d853ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5ckdt" Nov 03 20:29:45.187100 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.186653 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/09f023f0-38ef-4d16-a5d7-f3e309d853ba-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5ckdt\" (UID: \"09f023f0-38ef-4d16-a5d7-f3e309d853ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5ckdt" Nov 03 20:29:45.187100 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.186693 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09f023f0-38ef-4d16-a5d7-f3e309d853ba-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5ckdt\" (UID: \"09f023f0-38ef-4d16-a5d7-f3e309d853ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5ckdt" Nov 03 20:29:45.187698 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.187670 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09f023f0-38ef-4d16-a5d7-f3e309d853ba-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5ckdt\" (UID: \"09f023f0-38ef-4d16-a5d7-f3e309d853ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5ckdt" Nov 03 20:29:45.189248 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.189227 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09f023f0-38ef-4d16-a5d7-f3e309d853ba-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5ckdt\" (UID: \"09f023f0-38ef-4d16-a5d7-f3e309d853ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5ckdt" Nov 03 20:29:45.189358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.189233 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/09f023f0-38ef-4d16-a5d7-f3e309d853ba-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5ckdt\" (UID: \"09f023f0-38ef-4d16-a5d7-f3e309d853ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5ckdt" Nov 03 20:29:45.244459 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.244426 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx"] Nov 03 20:29:45.247136 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.247113 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx" Nov 03 20:29:45.251007 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.250983 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g9km\" (UniqueName: \"kubernetes.io/projected/09f023f0-38ef-4d16-a5d7-f3e309d853ba-kube-api-access-5g9km\") pod \"openshift-state-metrics-7bd7489957-5ckdt\" (UID: \"09f023f0-38ef-4d16-a5d7-f3e309d853ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5ckdt" Nov 03 20:29:45.257379 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.257300 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vjzhq"] Nov 03 20:29:45.259910 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.259888 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.288245 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.288225 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f779b3c3-bf33-44c8-8de5-cd75cece8e82-sys\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.288387 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.288255 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76800d8f-c9c9-45f3-8a2c-aa8f1c914026-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rlrxx\" (UID: \"76800d8f-c9c9-45f3-8a2c-aa8f1c914026\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx" Nov 03 20:29:45.288387 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.288276 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z2ss\" (UniqueName: \"kubernetes.io/projected/76800d8f-c9c9-45f3-8a2c-aa8f1c914026-kube-api-access-2z2ss\") pod \"kube-state-metrics-5dc8d7d89b-rlrxx\" (UID: \"76800d8f-c9c9-45f3-8a2c-aa8f1c914026\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx" Nov 03 20:29:45.288387 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.288299 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgrsq\" (UniqueName: \"kubernetes.io/projected/f779b3c3-bf33-44c8-8de5-cd75cece8e82-kube-api-access-jgrsq\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.288553 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.288476 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f779b3c3-bf33-44c8-8de5-cd75cece8e82-metrics-client-ca\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.288553 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.288514 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f779b3c3-bf33-44c8-8de5-cd75cece8e82-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.288553 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.288547 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f779b3c3-bf33-44c8-8de5-cd75cece8e82-root\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.288763 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.288569 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f779b3c3-bf33-44c8-8de5-cd75cece8e82-node-exporter-wtmp\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.288763 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.288608 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f779b3c3-bf33-44c8-8de5-cd75cece8e82-node-exporter-textfile\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.288763 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.288632 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f779b3c3-bf33-44c8-8de5-cd75cece8e82-node-exporter-tls\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.288763 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.288664 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/76800d8f-c9c9-45f3-8a2c-aa8f1c914026-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rlrxx\" (UID: \"76800d8f-c9c9-45f3-8a2c-aa8f1c914026\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx" Nov 03 20:29:45.288763 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.288707 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76800d8f-c9c9-45f3-8a2c-aa8f1c914026-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rlrxx\" (UID: \"76800d8f-c9c9-45f3-8a2c-aa8f1c914026\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx" Nov 03 20:29:45.288763 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.288747 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/76800d8f-c9c9-45f3-8a2c-aa8f1c914026-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rlrxx\" (UID: \"76800d8f-c9c9-45f3-8a2c-aa8f1c914026\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx" Nov 03 20:29:45.289025 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.288774 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f779b3c3-bf33-44c8-8de5-cd75cece8e82-node-exporter-accelerators-collector-config\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.289025 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.288813 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/76800d8f-c9c9-45f3-8a2c-aa8f1c914026-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rlrxx\" (UID: \"76800d8f-c9c9-45f3-8a2c-aa8f1c914026\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx" Nov 03 20:29:45.296656 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.296619 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 20:29:45.296764 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.296741 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-nff9l\"" Nov 03 20:29:45.296764 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.296743 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 20:29:45.296957 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.296938 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 20:29:45.303764 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.303742 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx"] Nov 03 20:29:45.321479 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.321460 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5ckdt" Nov 03 20:29:45.331526 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.326684 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 20:29:45.331526 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.326820 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2c9vt\"" Nov 03 20:29:45.331526 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.326919 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 20:29:45.390697 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.390320 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f779b3c3-bf33-44c8-8de5-cd75cece8e82-sys\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.390697 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.390391 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76800d8f-c9c9-45f3-8a2c-aa8f1c914026-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rlrxx\" (UID: \"76800d8f-c9c9-45f3-8a2c-aa8f1c914026\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx" Nov 03 20:29:45.390697 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.390428 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z2ss\" (UniqueName: \"kubernetes.io/projected/76800d8f-c9c9-45f3-8a2c-aa8f1c914026-kube-api-access-2z2ss\") pod \"kube-state-metrics-5dc8d7d89b-rlrxx\" (UID: \"76800d8f-c9c9-45f3-8a2c-aa8f1c914026\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx" Nov 03 20:29:45.390697 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.390482 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgrsq\" (UniqueName: \"kubernetes.io/projected/f779b3c3-bf33-44c8-8de5-cd75cece8e82-kube-api-access-jgrsq\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.390697 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.390515 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f779b3c3-bf33-44c8-8de5-cd75cece8e82-metrics-client-ca\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.390697 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.390540 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f779b3c3-bf33-44c8-8de5-cd75cece8e82-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.390697 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.390564 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f779b3c3-bf33-44c8-8de5-cd75cece8e82-root\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.390697 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.390587 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f779b3c3-bf33-44c8-8de5-cd75cece8e82-node-exporter-wtmp\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.390697 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.390625 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f779b3c3-bf33-44c8-8de5-cd75cece8e82-node-exporter-textfile\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.390697 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.390647 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f779b3c3-bf33-44c8-8de5-cd75cece8e82-node-exporter-tls\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.390697 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.390680 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/76800d8f-c9c9-45f3-8a2c-aa8f1c914026-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rlrxx\" (UID: \"76800d8f-c9c9-45f3-8a2c-aa8f1c914026\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx" Nov 03 20:29:45.391466 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.390722 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76800d8f-c9c9-45f3-8a2c-aa8f1c914026-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rlrxx\" (UID: \"76800d8f-c9c9-45f3-8a2c-aa8f1c914026\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx" Nov 03 20:29:45.391466 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.390778 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/76800d8f-c9c9-45f3-8a2c-aa8f1c914026-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rlrxx\" (UID: \"76800d8f-c9c9-45f3-8a2c-aa8f1c914026\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx" Nov 03 20:29:45.391466 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.390808 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f779b3c3-bf33-44c8-8de5-cd75cece8e82-node-exporter-accelerators-collector-config\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.391466 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.390847 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/76800d8f-c9c9-45f3-8a2c-aa8f1c914026-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rlrxx\" (UID: \"76800d8f-c9c9-45f3-8a2c-aa8f1c914026\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx" Nov 03 20:29:45.391861 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.391845 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/76800d8f-c9c9-45f3-8a2c-aa8f1c914026-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rlrxx\" (UID: \"76800d8f-c9c9-45f3-8a2c-aa8f1c914026\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx" Nov 03 20:29:45.392640 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.391923 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f779b3c3-bf33-44c8-8de5-cd75cece8e82-sys\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.392640 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.392431 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f779b3c3-bf33-44c8-8de5-cd75cece8e82-node-exporter-wtmp\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.393864 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.393713 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f779b3c3-bf33-44c8-8de5-cd75cece8e82-root\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.394427 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.394144 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/76800d8f-c9c9-45f3-8a2c-aa8f1c914026-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rlrxx\" (UID: \"76800d8f-c9c9-45f3-8a2c-aa8f1c914026\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx" Nov 03 20:29:45.394427 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.394163 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f779b3c3-bf33-44c8-8de5-cd75cece8e82-metrics-client-ca\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.395035 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.395011 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76800d8f-c9c9-45f3-8a2c-aa8f1c914026-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rlrxx\" (UID: \"76800d8f-c9c9-45f3-8a2c-aa8f1c914026\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx" Nov 03 20:29:45.395181 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.395157 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f779b3c3-bf33-44c8-8de5-cd75cece8e82-node-exporter-textfile\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.397566 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.397545 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 20:29:45.399418 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.399394 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f779b3c3-bf33-44c8-8de5-cd75cece8e82-node-exporter-tls\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.400869 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.400844 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f779b3c3-bf33-44c8-8de5-cd75cece8e82-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.401309 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.401285 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76800d8f-c9c9-45f3-8a2c-aa8f1c914026-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rlrxx\" (UID: \"76800d8f-c9c9-45f3-8a2c-aa8f1c914026\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx" Nov 03 20:29:45.403035 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.403012 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/76800d8f-c9c9-45f3-8a2c-aa8f1c914026-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rlrxx\" (UID: \"76800d8f-c9c9-45f3-8a2c-aa8f1c914026\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx" Nov 03 20:29:45.405217 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.405193 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f779b3c3-bf33-44c8-8de5-cd75cece8e82-node-exporter-accelerators-collector-config\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.447891 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.447865 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgrsq\" (UniqueName: \"kubernetes.io/projected/f779b3c3-bf33-44c8-8de5-cd75cece8e82-kube-api-access-jgrsq\") pod \"node-exporter-vjzhq\" (UID: \"f779b3c3-bf33-44c8-8de5-cd75cece8e82\") " pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.448043 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.448017 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z2ss\" (UniqueName: \"kubernetes.io/projected/76800d8f-c9c9-45f3-8a2c-aa8f1c914026-kube-api-access-2z2ss\") pod \"kube-state-metrics-5dc8d7d89b-rlrxx\" (UID: \"76800d8f-c9c9-45f3-8a2c-aa8f1c914026\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx" Nov 03 20:29:45.568936 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.568495 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx" Nov 03 20:29:45.575799 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.575771 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vjzhq" Nov 03 20:29:45.586114 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:29:45.586086 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf779b3c3_bf33_44c8_8de5_cd75cece8e82.slice/crio-221f5985d8a39755d5ea47f2749a0bd6214bc9f94239760e02060d2172e09f7f WatchSource:0}: Error finding container 221f5985d8a39755d5ea47f2749a0bd6214bc9f94239760e02060d2172e09f7f: Status 404 returned error can't find the container with id 221f5985d8a39755d5ea47f2749a0bd6214bc9f94239760e02060d2172e09f7f Nov 03 20:29:45.767272 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.767235 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-5ckdt"] Nov 03 20:29:45.774619 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:29:45.774584 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f023f0_38ef_4d16_a5d7_f3e309d853ba.slice/crio-97ce5e1718db3a8a978b20b325bd98b56ea874e1f5944947dffeff8c6029a191 WatchSource:0}: Error finding container 97ce5e1718db3a8a978b20b325bd98b56ea874e1f5944947dffeff8c6029a191: Status 404 returned error can't find the container with id 97ce5e1718db3a8a978b20b325bd98b56ea874e1f5944947dffeff8c6029a191 Nov 03 20:29:45.776695 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.776662 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vjzhq" event={"ID":"f779b3c3-bf33-44c8-8de5-cd75cece8e82","Type":"ContainerStarted","Data":"221f5985d8a39755d5ea47f2749a0bd6214bc9f94239760e02060d2172e09f7f"} Nov 03 20:29:45.961994 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:45.961959 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx"] Nov 03 20:29:45.969302 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:29:45.969276 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76800d8f_c9c9_45f3_8a2c_aa8f1c914026.slice/crio-db65092af029c70bd0d283e97da3edccb965bc092232719ed539833ec6a670df WatchSource:0}: Error finding container db65092af029c70bd0d283e97da3edccb965bc092232719ed539833ec6a670df: Status 404 returned error can't find the container with id db65092af029c70bd0d283e97da3edccb965bc092232719ed539833ec6a670df Nov 03 20:29:46.781277 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:46.781237 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx" event={"ID":"76800d8f-c9c9-45f3-8a2c-aa8f1c914026","Type":"ContainerStarted","Data":"db65092af029c70bd0d283e97da3edccb965bc092232719ed539833ec6a670df"} Nov 03 20:29:46.783791 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:46.783755 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5ckdt" event={"ID":"09f023f0-38ef-4d16-a5d7-f3e309d853ba","Type":"ContainerStarted","Data":"aeac46c4b7255ee3a57fef1c7fd1c84814901849537087aeb0ca012c22a72233"} Nov 03 20:29:46.783947 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:46.783816 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5ckdt" event={"ID":"09f023f0-38ef-4d16-a5d7-f3e309d853ba","Type":"ContainerStarted","Data":"29c3d4682b2b062e0642327718c3a540649f0b204bbccaed10f92d5eac735e4b"} Nov 03 20:29:46.783947 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:46.783844 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5ckdt" event={"ID":"09f023f0-38ef-4d16-a5d7-f3e309d853ba","Type":"ContainerStarted","Data":"97ce5e1718db3a8a978b20b325bd98b56ea874e1f5944947dffeff8c6029a191"} Nov 03 20:29:47.790280 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:47.790242 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5ckdt" event={"ID":"09f023f0-38ef-4d16-a5d7-f3e309d853ba","Type":"ContainerStarted","Data":"377dacdd7f3e7a0b7f7ec8991188008ca1f26587342fcf582f621f3c3d08b5fb"} Nov 03 20:29:47.792530 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:47.792480 4493 generic.go:356] "Generic (PLEG): container finished" podID="f779b3c3-bf33-44c8-8de5-cd75cece8e82" containerID="64e114f39e53025e4f46e250baea846a300b0c25d4c4e2bd0669cf678435367c" exitCode=0 Nov 03 20:29:47.792650 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:47.792531 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vjzhq" event={"ID":"f779b3c3-bf33-44c8-8de5-cd75cece8e82","Type":"ContainerDied","Data":"64e114f39e53025e4f46e250baea846a300b0c25d4c4e2bd0669cf678435367c"} Nov 03 20:29:47.934965 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:47.934918 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5ckdt" podStartSLOduration=2.304571663 podStartE2EDuration="3.934897558s" totalImagesPullingTime="1.630325895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:47.934508648 +0000 UTC m=+59.492499148" watchObservedRunningTime="2026-11-03 20:29:47.934897558 +0000 UTC m=+59.492888061" Nov 03 20:29:48.798296 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:48.798258 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vjzhq" event={"ID":"f779b3c3-bf33-44c8-8de5-cd75cece8e82","Type":"ContainerStarted","Data":"ecc1d5285c93503d8da693021c9d72d3fe01fe3f3af65873776b389745cafb67"} Nov 03 20:29:48.798764 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:48.798321 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vjzhq" event={"ID":"f779b3c3-bf33-44c8-8de5-cd75cece8e82","Type":"ContainerStarted","Data":"d3469e008713cc5207f956a81a0a8f7840b11c663c61dc305661205904f48cde"} Nov 03 20:29:48.800523 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:48.800496 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx" event={"ID":"76800d8f-c9c9-45f3-8a2c-aa8f1c914026","Type":"ContainerStarted","Data":"2615325a7aaf162c83cf05afda9d79c107e2b85067fafcfad733cca017af8f1e"} Nov 03 20:29:48.800647 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:48.800554 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx" event={"ID":"76800d8f-c9c9-45f3-8a2c-aa8f1c914026","Type":"ContainerStarted","Data":"7643624006b6c49f3644db69821dee7b62706eb7435ecea7390ec07728cf383b"} Nov 03 20:29:48.800647 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:48.800573 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx" event={"ID":"76800d8f-c9c9-45f3-8a2c-aa8f1c914026","Type":"ContainerStarted","Data":"a4b65fbda036cfbcab0258db84b3c3c53a2bee2a44607cabb71035acbcb95c7a"} Nov 03 20:29:48.907882 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:48.907836 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vjzhq" podStartSLOduration=2.222959794 podStartE2EDuration="3.907821655s" totalImagesPullingTime="1.586883656s" totalInitContainerRuntime="97.978205ms" isStatefulPod=true podCreationTimestamp="2026-11-03 20:29:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:48.906638529 +0000 UTC m=+60.464629029" watchObservedRunningTime="2026-11-03 20:29:48.907821655 +0000 UTC m=+60.465812146" Nov 03 20:29:49.016882 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:49.016820 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rlrxx" podStartSLOduration=2.068280406 podStartE2EDuration="4.016800761s" totalImagesPullingTime="1.948520355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:49.015957677 +0000 UTC m=+60.573948197" watchObservedRunningTime="2026-11-03 20:29:49.016800761 +0000 UTC m=+60.574791263" Nov 03 20:29:51.742972 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:29:51.742935 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:29:54.235493 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.235457 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:29:54.241443 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.241418 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.291464 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.291440 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 20:29:54.291464 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.291438 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 20:29:54.339309 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.339075 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 20:29:54.339309 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.339081 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 20:29:54.339544 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.339350 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 20:29:54.339544 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.339438 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 20:29:54.339544 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.339536 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e4gqi9de47aks\"" Nov 03 20:29:54.339887 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.339864 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 20:29:54.342082 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.342057 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:29:54.359852 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.359819 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.359988 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.359862 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.359988 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.359908 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.360080 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.359995 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.360080 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.360029 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ea6ce6d-cda8-4676-a237-01e667f6815a-config-out\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.360159 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.360091 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.360159 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.360122 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.360159 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.360143 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.360265 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.360164 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-web-config\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.360265 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.360203 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.360265 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.360255 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-config\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.360380 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.360273 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.360380 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.360291 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.360380 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.360314 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.360380 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.360371 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4czw9\" (UniqueName: \"kubernetes.io/projected/6ea6ce6d-cda8-4676-a237-01e667f6815a-kube-api-access-4czw9\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.360509 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.360430 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.360509 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.360459 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.360509 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.360481 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ea6ce6d-cda8-4676-a237-01e667f6815a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.360609 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.360519 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.360609 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.360542 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.391830 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.391795 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jrwnj\"" Nov 03 20:29:54.391830 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.391804 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 20:29:54.392047 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.391843 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 20:29:54.412399 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.412366 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 20:29:54.412399 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.412393 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 20:29:54.412560 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.412398 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 20:29:54.412560 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.412401 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 20:29:54.460467 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.460437 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 20:29:54.460467 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.460457 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 20:29:54.461599 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.461579 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.461696 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.461611 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.461696 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.461654 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.461696 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.461682 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.461842 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.461712 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ea6ce6d-cda8-4676-a237-01e667f6815a-config-out\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.461842 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.461751 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.461934 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.461883 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.461986 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.461930 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.461986 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.461958 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-web-config\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.462080 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.461985 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.462080 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.462016 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-config\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.462080 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.462045 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.462080 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.462073 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.462292 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.462126 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.462292 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.462153 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4czw9\" (UniqueName: \"kubernetes.io/projected/6ea6ce6d-cda8-4676-a237-01e667f6815a-kube-api-access-4czw9\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.462292 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.462183 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.462292 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.462216 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.462292 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.462240 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ea6ce6d-cda8-4676-a237-01e667f6815a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.462627 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.462601 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.464197 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.462180 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.464197 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.463275 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.464197 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.463582 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.464197 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.463720 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.464197 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.463771 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.464197 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.463797 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.464197 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.463913 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.465742 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.465718 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.466155 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.466129 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.466244 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.466129 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.466536 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.466512 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.467007 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.466981 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.467119 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.467097 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.467195 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.467174 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ea6ce6d-cda8-4676-a237-01e667f6815a-config-out\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.467530 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.467510 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.467679 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.467662 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-web-config\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.468029 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.468016 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ea6ce6d-cda8-4676-a237-01e667f6815a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.468510 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.468488 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-config\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.468777 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.468756 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.468851 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.468765 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.516056 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.515980 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4czw9\" (UniqueName: \"kubernetes.io/projected/6ea6ce6d-cda8-4676-a237-01e667f6815a-kube-api-access-4czw9\") pod \"prometheus-k8s-0\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.569002 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.568977 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:29:54.994130 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:54.994098 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:29:55.000764 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:29:55.000737 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea6ce6d_cda8_4676_a237_01e667f6815a.slice/crio-ceb59aba2dcb60d4033cbbbefde8d33bf82466890c3e3b8fd864758ae504416b WatchSource:0}: Error finding container ceb59aba2dcb60d4033cbbbefde8d33bf82466890c3e3b8fd864758ae504416b: Status 404 returned error can't find the container with id ceb59aba2dcb60d4033cbbbefde8d33bf82466890c3e3b8fd864758ae504416b Nov 03 20:29:55.824977 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:55.824941 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ea6ce6d-cda8-4676-a237-01e667f6815a","Type":"ContainerStarted","Data":"ceb59aba2dcb60d4033cbbbefde8d33bf82466890c3e3b8fd864758ae504416b"} Nov 03 20:29:56.830154 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:56.830120 4493 generic.go:356] "Generic (PLEG): container finished" podID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerID="da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40" exitCode=0 Nov 03 20:29:56.830579 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:29:56.830205 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ea6ce6d-cda8-4676-a237-01e667f6815a","Type":"ContainerDied","Data":"da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40"} Nov 03 20:30:00.849249 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:00.849211 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ea6ce6d-cda8-4676-a237-01e667f6815a","Type":"ContainerStarted","Data":"066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea"} Nov 03 20:30:00.849672 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:00.849283 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ea6ce6d-cda8-4676-a237-01e667f6815a","Type":"ContainerStarted","Data":"fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2"} Nov 03 20:30:03.864216 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:03.864181 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ea6ce6d-cda8-4676-a237-01e667f6815a","Type":"ContainerStarted","Data":"f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b"} Nov 03 20:30:03.864680 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:03.864292 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ea6ce6d-cda8-4676-a237-01e667f6815a","Type":"ContainerStarted","Data":"a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179"} Nov 03 20:30:03.864680 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:03.864362 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ea6ce6d-cda8-4676-a237-01e667f6815a","Type":"ContainerStarted","Data":"58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d"} Nov 03 20:30:03.864680 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:03.864401 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ea6ce6d-cda8-4676-a237-01e667f6815a","Type":"ContainerStarted","Data":"9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376"} Nov 03 20:30:03.987615 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:03.987565 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.183375566 podStartE2EDuration="9.987551063s" totalImagesPullingTime="7.783897374s" totalInitContainerRuntime="20.278123ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:03.986895998 +0000 UTC m=+75.544886497" watchObservedRunningTime="2026-11-03 20:30:03.987551063 +0000 UTC m=+75.545541583" Nov 03 20:30:04.569394 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:04.569358 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:30:09.200225 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.200161 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" podUID="2296e3c6-6709-4fb0-8384-b0800d874d32" containerName="registry" containerID="cri-o://1dcfed70aa4685cb24bd7042f86afcb11facd2af48147910c1168142d34f899f" gracePeriod=30 Nov 03 20:30:09.444543 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.444520 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:30:09.606173 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.606140 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2296e3c6-6709-4fb0-8384-b0800d874d32-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2296e3c6-6709-4fb0-8384-b0800d874d32-bound-sa-token\") pod \"2296e3c6-6709-4fb0-8384-b0800d874d32\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " Nov 03 20:30:09.606399 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.606188 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2296e3c6-6709-4fb0-8384-b0800d874d32-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2296e3c6-6709-4fb0-8384-b0800d874d32-ca-trust-extracted\") pod \"2296e3c6-6709-4fb0-8384-b0800d874d32\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " Nov 03 20:30:09.606399 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.606231 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2296e3c6-6709-4fb0-8384-b0800d874d32-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2296e3c6-6709-4fb0-8384-b0800d874d32-image-registry-private-configuration\") pod \"2296e3c6-6709-4fb0-8384-b0800d874d32\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " Nov 03 20:30:09.606399 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.606280 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2296e3c6-6709-4fb0-8384-b0800d874d32-kube-api-access-vsfrj\" (UniqueName: \"kubernetes.io/projected/2296e3c6-6709-4fb0-8384-b0800d874d32-kube-api-access-vsfrj\") pod \"2296e3c6-6709-4fb0-8384-b0800d874d32\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " Nov 03 20:30:09.606399 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.606307 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2296e3c6-6709-4fb0-8384-b0800d874d32-registry-tls\" (UniqueName: \"kubernetes.io/projected/2296e3c6-6709-4fb0-8384-b0800d874d32-registry-tls\") pod \"2296e3c6-6709-4fb0-8384-b0800d874d32\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " Nov 03 20:30:09.606399 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.606363 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2296e3c6-6709-4fb0-8384-b0800d874d32-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2296e3c6-6709-4fb0-8384-b0800d874d32-trusted-ca\") pod \"2296e3c6-6709-4fb0-8384-b0800d874d32\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " Nov 03 20:30:09.606399 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.606396 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2296e3c6-6709-4fb0-8384-b0800d874d32-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2296e3c6-6709-4fb0-8384-b0800d874d32-installation-pull-secrets\") pod \"2296e3c6-6709-4fb0-8384-b0800d874d32\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " Nov 03 20:30:09.607018 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.606441 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2296e3c6-6709-4fb0-8384-b0800d874d32-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2296e3c6-6709-4fb0-8384-b0800d874d32-registry-certificates\") pod \"2296e3c6-6709-4fb0-8384-b0800d874d32\" (UID: \"2296e3c6-6709-4fb0-8384-b0800d874d32\") " Nov 03 20:30:09.607113 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.607086 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2296e3c6-6709-4fb0-8384-b0800d874d32-registry-certificates" pod "2296e3c6-6709-4fb0-8384-b0800d874d32" (UID: "2296e3c6-6709-4fb0-8384-b0800d874d32"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:09.607177 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.607151 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2296e3c6-6709-4fb0-8384-b0800d874d32-trusted-ca" pod "2296e3c6-6709-4fb0-8384-b0800d874d32" (UID: "2296e3c6-6709-4fb0-8384-b0800d874d32"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:09.610480 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.610447 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2296e3c6-6709-4fb0-8384-b0800d874d32-installation-pull-secrets" pod "2296e3c6-6709-4fb0-8384-b0800d874d32" (UID: "2296e3c6-6709-4fb0-8384-b0800d874d32"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:30:09.610480 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.610460 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2296e3c6-6709-4fb0-8384-b0800d874d32-registry-tls" pod "2296e3c6-6709-4fb0-8384-b0800d874d32" (UID: "2296e3c6-6709-4fb0-8384-b0800d874d32"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:09.610638 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.610480 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2296e3c6-6709-4fb0-8384-b0800d874d32-kube-api-access-vsfrj" pod "2296e3c6-6709-4fb0-8384-b0800d874d32" (UID: "2296e3c6-6709-4fb0-8384-b0800d874d32"). InnerVolumeSpecName "kube-api-access-vsfrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:09.610638 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.610494 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2296e3c6-6709-4fb0-8384-b0800d874d32-bound-sa-token" pod "2296e3c6-6709-4fb0-8384-b0800d874d32" (UID: "2296e3c6-6709-4fb0-8384-b0800d874d32"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:09.611014 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.610984 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2296e3c6-6709-4fb0-8384-b0800d874d32-image-registry-private-configuration" pod "2296e3c6-6709-4fb0-8384-b0800d874d32" (UID: "2296e3c6-6709-4fb0-8384-b0800d874d32"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:30:09.615485 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.615458 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2296e3c6-6709-4fb0-8384-b0800d874d32-ca-trust-extracted" pod "2296e3c6-6709-4fb0-8384-b0800d874d32" (UID: "2296e3c6-6709-4fb0-8384-b0800d874d32"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:09.707317 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.707290 4493 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2296e3c6-6709-4fb0-8384-b0800d874d32-bound-sa-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:09.707317 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.707316 4493 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2296e3c6-6709-4fb0-8384-b0800d874d32-ca-trust-extracted\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:09.707506 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.707327 4493 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2296e3c6-6709-4fb0-8384-b0800d874d32-image-registry-private-configuration\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:09.707506 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.707354 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsfrj\" (UniqueName: \"kubernetes.io/projected/2296e3c6-6709-4fb0-8384-b0800d874d32-kube-api-access-vsfrj\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:09.707506 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.707366 4493 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2296e3c6-6709-4fb0-8384-b0800d874d32-registry-tls\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:09.707506 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.707375 4493 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2296e3c6-6709-4fb0-8384-b0800d874d32-trusted-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:09.707506 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.707383 4493 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2296e3c6-6709-4fb0-8384-b0800d874d32-installation-pull-secrets\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:09.707506 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.707391 4493 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2296e3c6-6709-4fb0-8384-b0800d874d32-registry-certificates\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:09.886245 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.886153 4493 generic.go:356] "Generic (PLEG): container finished" podID="2296e3c6-6709-4fb0-8384-b0800d874d32" containerID="1dcfed70aa4685cb24bd7042f86afcb11facd2af48147910c1168142d34f899f" exitCode=0 Nov 03 20:30:09.886245 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.886206 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" event={"ID":"2296e3c6-6709-4fb0-8384-b0800d874d32","Type":"ContainerDied","Data":"1dcfed70aa4685cb24bd7042f86afcb11facd2af48147910c1168142d34f899f"} Nov 03 20:30:09.886245 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.886209 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" Nov 03 20:30:09.886245 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.886247 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65c4b8fd96-jpckm" event={"ID":"2296e3c6-6709-4fb0-8384-b0800d874d32","Type":"ContainerDied","Data":"7b707c0e0bb949bf2afc47ea2907950196440e8b2fd16d0a74a14cf83d74b9a2"} Nov 03 20:30:09.886595 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.886266 4493 scope.go:122] "RemoveContainer" containerID="1dcfed70aa4685cb24bd7042f86afcb11facd2af48147910c1168142d34f899f" Nov 03 20:30:09.899135 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.899116 4493 scope.go:122] "RemoveContainer" containerID="1dcfed70aa4685cb24bd7042f86afcb11facd2af48147910c1168142d34f899f" Nov 03 20:30:09.899431 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:30:09.899411 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dcfed70aa4685cb24bd7042f86afcb11facd2af48147910c1168142d34f899f\": container with ID starting with 1dcfed70aa4685cb24bd7042f86afcb11facd2af48147910c1168142d34f899f not found: ID does not exist" containerID="1dcfed70aa4685cb24bd7042f86afcb11facd2af48147910c1168142d34f899f" Nov 03 20:30:09.899491 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:09.899441 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dcfed70aa4685cb24bd7042f86afcb11facd2af48147910c1168142d34f899f"} err="failed to get container status \"1dcfed70aa4685cb24bd7042f86afcb11facd2af48147910c1168142d34f899f\": rpc error: code = NotFound desc = could not find container \"1dcfed70aa4685cb24bd7042f86afcb11facd2af48147910c1168142d34f899f\": container with ID starting with 1dcfed70aa4685cb24bd7042f86afcb11facd2af48147910c1168142d34f899f not found: ID does not exist" Nov 03 20:30:10.040823 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:10.040794 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65c4b8fd96-jpckm"] Nov 03 20:30:10.087661 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:10.087623 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-65c4b8fd96-jpckm"] Nov 03 20:30:10.559031 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:10.559000 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2296e3c6-6709-4fb0-8384-b0800d874d32" path="/var/lib/kubelet/pods/2296e3c6-6709-4fb0-8384-b0800d874d32/volumes" Nov 03 20:30:21.743383 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:30:21.743321 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:30:50.020627 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:50.020590 4493 generic.go:356] "Generic (PLEG): container finished" podID="08c9d7c8-dbd8-46ad-92f4-2a009e83abc3" containerID="a054fcf5f233995135e99c64f3fbff1dec8e78e45b1033764e47cc27ba1f528e" exitCode=0 Nov 03 20:30:50.021060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:50.020650 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wrlml-k5rcz" event={"ID":"08c9d7c8-dbd8-46ad-92f4-2a009e83abc3","Type":"ContainerDied","Data":"a054fcf5f233995135e99c64f3fbff1dec8e78e45b1033764e47cc27ba1f528e"} Nov 03 20:30:51.158079 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:51.158055 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wrlml-k5rcz" Nov 03 20:30:51.209955 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:51.209923 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-trusted-ca-bundle\") pod \"08c9d7c8-dbd8-46ad-92f4-2a009e83abc3\" (UID: \"08c9d7c8-dbd8-46ad-92f4-2a009e83abc3\") " Nov 03 20:30:51.210136 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:51.209974 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-archives-path\") pod \"08c9d7c8-dbd8-46ad-92f4-2a009e83abc3\" (UID: \"08c9d7c8-dbd8-46ad-92f4-2a009e83abc3\") " Nov 03 20:30:51.210136 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:51.210006 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-service-ca-bundle\") pod \"08c9d7c8-dbd8-46ad-92f4-2a009e83abc3\" (UID: \"08c9d7c8-dbd8-46ad-92f4-2a009e83abc3\") " Nov 03 20:30:51.210136 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:51.210024 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-kube-api-access-xqrdc\" (UniqueName: \"kubernetes.io/projected/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-kube-api-access-xqrdc\") pod \"08c9d7c8-dbd8-46ad-92f4-2a009e83abc3\" (UID: \"08c9d7c8-dbd8-46ad-92f4-2a009e83abc3\") " Nov 03 20:30:51.210355 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:51.210307 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-archives-path" pod "08c9d7c8-dbd8-46ad-92f4-2a009e83abc3" (UID: "08c9d7c8-dbd8-46ad-92f4-2a009e83abc3"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:51.210416 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:51.210366 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-service-ca-bundle" pod "08c9d7c8-dbd8-46ad-92f4-2a009e83abc3" (UID: "08c9d7c8-dbd8-46ad-92f4-2a009e83abc3"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:51.210416 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:51.210397 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-trusted-ca-bundle" pod "08c9d7c8-dbd8-46ad-92f4-2a009e83abc3" (UID: "08c9d7c8-dbd8-46ad-92f4-2a009e83abc3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:51.212183 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:51.212165 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-kube-api-access-xqrdc" pod "08c9d7c8-dbd8-46ad-92f4-2a009e83abc3" (UID: "08c9d7c8-dbd8-46ad-92f4-2a009e83abc3"). InnerVolumeSpecName "kube-api-access-xqrdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:51.311043 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:51.311017 4493 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-trusted-ca-bundle\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:51.311043 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:51.311041 4493 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-archives-path\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:51.311197 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:51.311050 4493 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-service-ca-bundle\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:51.311197 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:51.311059 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqrdc\" (UniqueName: \"kubernetes.io/projected/08c9d7c8-dbd8-46ad-92f4-2a009e83abc3-kube-api-access-xqrdc\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:51.742826 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:30:51.742751 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:30:52.028878 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:52.028799 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wrlml-k5rcz" event={"ID":"08c9d7c8-dbd8-46ad-92f4-2a009e83abc3","Type":"ContainerDied","Data":"4fa25eaa7a3420c901cd2058dccc33d4869a9255668c691a5fa2b6f4a8d57e4c"} Nov 03 20:30:52.028878 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:52.028836 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wrlml-k5rcz" Nov 03 20:30:52.029053 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:52.028840 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fa25eaa7a3420c901cd2058dccc33d4869a9255668c691a5fa2b6f4a8d57e4c" Nov 03 20:30:54.570057 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:54.570022 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:30:54.589269 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:54.589246 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:30:55.053983 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:30:55.053958 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:17.632012 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:17.631967 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:31:17.633857 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:17.633836 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerName="prometheus" containerID="cri-o://fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2" gracePeriod=600 Nov 03 20:31:17.634015 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:17.633994 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerName="kube-rbac-proxy-web" containerID="cri-o://58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d" gracePeriod=600 Nov 03 20:31:17.634247 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:17.634056 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerName="config-reloader" containerID="cri-o://066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea" gracePeriod=600 Nov 03 20:31:17.634247 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:17.634113 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerName="thanos-sidecar" containerID="cri-o://9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376" gracePeriod=600 Nov 03 20:31:17.634247 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:17.634055 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerName="kube-rbac-proxy-thanos" containerID="cri-o://f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b" gracePeriod=600 Nov 03 20:31:17.634247 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:17.633997 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerName="kube-rbac-proxy" containerID="cri-o://a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179" gracePeriod=600 Nov 03 20:31:17.888146 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:17.888091 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.021717 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.021690 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-trusted-ca-bundle\") pod \"6ea6ce6d-cda8-4676-a237-01e667f6815a\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " Nov 03 20:31:18.021886 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.021724 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ea6ce6d-cda8-4676-a237-01e667f6815a-tls-assets\" (UniqueName: \"kubernetes.io/projected/6ea6ce6d-cda8-4676-a237-01e667f6815a-tls-assets\") pod \"6ea6ce6d-cda8-4676-a237-01e667f6815a\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " Nov 03 20:31:18.021886 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.021753 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ea6ce6d-cda8-4676-a237-01e667f6815a-kube-api-access-4czw9\" (UniqueName: \"kubernetes.io/projected/6ea6ce6d-cda8-4676-a237-01e667f6815a-kube-api-access-4czw9\") pod \"6ea6ce6d-cda8-4676-a237-01e667f6815a\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " Nov 03 20:31:18.021886 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.021781 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-rulefiles-2\") pod \"6ea6ce6d-cda8-4676-a237-01e667f6815a\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " Nov 03 20:31:18.021886 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.021810 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-db\") pod \"6ea6ce6d-cda8-4676-a237-01e667f6815a\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " Nov 03 20:31:18.021886 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.021845 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-config\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-config\") pod \"6ea6ce6d-cda8-4676-a237-01e667f6815a\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " Nov 03 20:31:18.021886 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.021884 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea6ce6d-cda8-4676-a237-01e667f6815a-config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ea6ce6d-cda8-4676-a237-01e667f6815a-config-out\") pod \"6ea6ce6d-cda8-4676-a237-01e667f6815a\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " Nov 03 20:31:18.022191 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.021923 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-thanos-prometheus-http-client-file\") pod \"6ea6ce6d-cda8-4676-a237-01e667f6815a\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " Nov 03 20:31:18.022191 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.021963 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-grpc-tls\") pod \"6ea6ce6d-cda8-4676-a237-01e667f6815a\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " Nov 03 20:31:18.022191 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.021995 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-rulefiles-1\") pod \"6ea6ce6d-cda8-4676-a237-01e667f6815a\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " Nov 03 20:31:18.022191 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.022027 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-web-config\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-web-config\") pod \"6ea6ce6d-cda8-4676-a237-01e667f6815a\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " Nov 03 20:31:18.022191 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.022078 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-rulefiles-0\") pod \"6ea6ce6d-cda8-4676-a237-01e667f6815a\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " Nov 03 20:31:18.022191 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.022113 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-configmap-kubelet-serving-ca-bundle\") pod \"6ea6ce6d-cda8-4676-a237-01e667f6815a\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " Nov 03 20:31:18.022191 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.022133 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-trusted-ca-bundle" pod "6ea6ce6d-cda8-4676-a237-01e667f6815a" (UID: "6ea6ce6d-cda8-4676-a237-01e667f6815a"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:18.022191 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.022167 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"6ea6ce6d-cda8-4676-a237-01e667f6815a\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " Nov 03 20:31:18.022649 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.022207 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-configmap-metrics-client-ca\") pod \"6ea6ce6d-cda8-4676-a237-01e667f6815a\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " Nov 03 20:31:18.022649 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.022217 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-rulefiles-2" pod "6ea6ce6d-cda8-4676-a237-01e667f6815a" (UID: "6ea6ce6d-cda8-4676-a237-01e667f6815a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:18.022649 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.022236 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-metrics-client-certs\") pod \"6ea6ce6d-cda8-4676-a237-01e667f6815a\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " Nov 03 20:31:18.022649 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.022271 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-prometheus-k8s-tls\") pod \"6ea6ce6d-cda8-4676-a237-01e667f6815a\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " Nov 03 20:31:18.022649 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.022311 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-kube-rbac-proxy\") pod \"6ea6ce6d-cda8-4676-a237-01e667f6815a\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " Nov 03 20:31:18.022649 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.022369 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-configmap-serving-certs-ca-bundle\") pod \"6ea6ce6d-cda8-4676-a237-01e667f6815a\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " Nov 03 20:31:18.022649 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.022399 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"6ea6ce6d-cda8-4676-a237-01e667f6815a\" (UID: \"6ea6ce6d-cda8-4676-a237-01e667f6815a\") " Nov 03 20:31:18.022649 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.022645 4493 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:18.023201 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.022663 4493 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:18.024835 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.024669 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-rulefiles-1" pod "6ea6ce6d-cda8-4676-a237-01e667f6815a" (UID: "6ea6ce6d-cda8-4676-a237-01e667f6815a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:18.025276 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.025168 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-rulefiles-0" pod "6ea6ce6d-cda8-4676-a237-01e667f6815a" (UID: "6ea6ce6d-cda8-4676-a237-01e667f6815a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:18.025523 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.025483 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-configmap-metrics-client-ca" pod "6ea6ce6d-cda8-4676-a237-01e667f6815a" (UID: "6ea6ce6d-cda8-4676-a237-01e667f6815a"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:18.025849 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.025794 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-db" pod "6ea6ce6d-cda8-4676-a237-01e667f6815a" (UID: "6ea6ce6d-cda8-4676-a237-01e667f6815a"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:18.025986 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.025944 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-configmap-serving-certs-ca-bundle" pod "6ea6ce6d-cda8-4676-a237-01e667f6815a" (UID: "6ea6ce6d-cda8-4676-a237-01e667f6815a"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:18.026093 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.025990 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-configmap-kubelet-serving-ca-bundle" pod "6ea6ce6d-cda8-4676-a237-01e667f6815a" (UID: "6ea6ce6d-cda8-4676-a237-01e667f6815a"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:18.026321 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.026303 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-prometheus-k8s-thanos-sidecar-tls" pod "6ea6ce6d-cda8-4676-a237-01e667f6815a" (UID: "6ea6ce6d-cda8-4676-a237-01e667f6815a"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:18.027248 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.027221 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-grpc-tls" pod "6ea6ce6d-cda8-4676-a237-01e667f6815a" (UID: "6ea6ce6d-cda8-4676-a237-01e667f6815a"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:18.027864 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.027824 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-config" pod "6ea6ce6d-cda8-4676-a237-01e667f6815a" (UID: "6ea6ce6d-cda8-4676-a237-01e667f6815a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:18.027965 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.027927 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-thanos-prometheus-http-client-file" pod "6ea6ce6d-cda8-4676-a237-01e667f6815a" (UID: "6ea6ce6d-cda8-4676-a237-01e667f6815a"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:18.028664 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.028640 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-prometheus-k8s-tls" pod "6ea6ce6d-cda8-4676-a237-01e667f6815a" (UID: "6ea6ce6d-cda8-4676-a237-01e667f6815a"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:18.028755 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.028663 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-prometheus-k8s-kube-rbac-proxy-web" pod "6ea6ce6d-cda8-4676-a237-01e667f6815a" (UID: "6ea6ce6d-cda8-4676-a237-01e667f6815a"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:18.028755 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.028723 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea6ce6d-cda8-4676-a237-01e667f6815a-tls-assets" pod "6ea6ce6d-cda8-4676-a237-01e667f6815a" (UID: "6ea6ce6d-cda8-4676-a237-01e667f6815a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:18.028829 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.028748 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea6ce6d-cda8-4676-a237-01e667f6815a-kube-api-access-4czw9" pod "6ea6ce6d-cda8-4676-a237-01e667f6815a" (UID: "6ea6ce6d-cda8-4676-a237-01e667f6815a"). InnerVolumeSpecName "kube-api-access-4czw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:18.029275 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.029236 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea6ce6d-cda8-4676-a237-01e667f6815a-config-out" pod "6ea6ce6d-cda8-4676-a237-01e667f6815a" (UID: "6ea6ce6d-cda8-4676-a237-01e667f6815a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:18.029472 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.029455 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-metrics-client-certs" pod "6ea6ce6d-cda8-4676-a237-01e667f6815a" (UID: "6ea6ce6d-cda8-4676-a237-01e667f6815a"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:18.030608 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.030577 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-kube-rbac-proxy" pod "6ea6ce6d-cda8-4676-a237-01e667f6815a" (UID: "6ea6ce6d-cda8-4676-a237-01e667f6815a"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:18.037772 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.037750 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-web-config" pod "6ea6ce6d-cda8-4676-a237-01e667f6815a" (UID: "6ea6ce6d-cda8-4676-a237-01e667f6815a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:18.117770 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.117737 4493 generic.go:356] "Generic (PLEG): container finished" podID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerID="f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b" exitCode=0 Nov 03 20:31:18.117770 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.117765 4493 generic.go:356] "Generic (PLEG): container finished" podID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerID="a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179" exitCode=0 Nov 03 20:31:18.117770 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.117772 4493 generic.go:356] "Generic (PLEG): container finished" podID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerID="58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d" exitCode=0 Nov 03 20:31:18.117952 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.117779 4493 generic.go:356] "Generic (PLEG): container finished" podID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerID="9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376" exitCode=0 Nov 03 20:31:18.117952 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.117785 4493 generic.go:356] "Generic (PLEG): container finished" podID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerID="066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea" exitCode=0 Nov 03 20:31:18.117952 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.117791 4493 generic.go:356] "Generic (PLEG): container finished" podID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerID="fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2" exitCode=0 Nov 03 20:31:18.117952 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.117813 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.117952 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.117821 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ea6ce6d-cda8-4676-a237-01e667f6815a","Type":"ContainerDied","Data":"f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b"} Nov 03 20:31:18.117952 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.117912 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ea6ce6d-cda8-4676-a237-01e667f6815a","Type":"ContainerDied","Data":"a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179"} Nov 03 20:31:18.117952 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.117939 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ea6ce6d-cda8-4676-a237-01e667f6815a","Type":"ContainerDied","Data":"58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d"} Nov 03 20:31:18.118165 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.117964 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ea6ce6d-cda8-4676-a237-01e667f6815a","Type":"ContainerDied","Data":"9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376"} Nov 03 20:31:18.118165 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.117986 4493 scope.go:122] "RemoveContainer" containerID="f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b" Nov 03 20:31:18.118165 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.118002 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ea6ce6d-cda8-4676-a237-01e667f6815a","Type":"ContainerDied","Data":"066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea"} Nov 03 20:31:18.118165 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.118026 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ea6ce6d-cda8-4676-a237-01e667f6815a","Type":"ContainerDied","Data":"fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2"} Nov 03 20:31:18.118165 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.118050 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ea6ce6d-cda8-4676-a237-01e667f6815a","Type":"ContainerDied","Data":"ceb59aba2dcb60d4033cbbbefde8d33bf82466890c3e3b8fd864758ae504416b"} Nov 03 20:31:18.123351 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.123313 4493 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-grpc-tls\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:18.123447 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.123359 4493 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:18.123447 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.123376 4493 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-web-config\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:18.123447 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.123392 4493 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:18.123447 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.123408 4493 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:18.123447 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.123424 4493 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:18.123447 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.123439 4493 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-configmap-metrics-client-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:18.123631 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.123454 4493 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-metrics-client-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:18.123631 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.123468 4493 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:18.123631 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.123485 4493 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-kube-rbac-proxy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:18.123631 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.123499 4493 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ea6ce6d-cda8-4676-a237-01e667f6815a-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:18.123631 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.123515 4493 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:18.123631 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.123531 4493 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ea6ce6d-cda8-4676-a237-01e667f6815a-tls-assets\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:18.123631 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.123545 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4czw9\" (UniqueName: \"kubernetes.io/projected/6ea6ce6d-cda8-4676-a237-01e667f6815a-kube-api-access-4czw9\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:18.123631 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.123560 4493 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6ea6ce6d-cda8-4676-a237-01e667f6815a-prometheus-k8s-db\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:18.123631 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.123574 4493 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-config\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:18.123631 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.123587 4493 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ea6ce6d-cda8-4676-a237-01e667f6815a-config-out\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:18.123631 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.123603 4493 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6ea6ce6d-cda8-4676-a237-01e667f6815a-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:18.125711 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.125635 4493 scope.go:122] "RemoveContainer" containerID="a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179" Nov 03 20:31:18.134381 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.134359 4493 scope.go:122] "RemoveContainer" containerID="58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d" Nov 03 20:31:18.141289 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.141269 4493 scope.go:122] "RemoveContainer" containerID="9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376" Nov 03 20:31:18.147878 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.147862 4493 scope.go:122] "RemoveContainer" containerID="066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea" Nov 03 20:31:18.154079 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.154062 4493 scope.go:122] "RemoveContainer" containerID="fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2" Nov 03 20:31:18.160777 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.160758 4493 scope.go:122] "RemoveContainer" containerID="da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40" Nov 03 20:31:18.166985 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.166966 4493 scope.go:122] "RemoveContainer" containerID="f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b" Nov 03 20:31:18.167272 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:31:18.167251 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b\": container with ID starting with f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b not found: ID does not exist" containerID="f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b" Nov 03 20:31:18.167387 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.167282 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b"} err="failed to get container status \"f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b\": rpc error: code = NotFound desc = could not find container \"f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b\": container with ID starting with f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b not found: ID does not exist" Nov 03 20:31:18.167387 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.167302 4493 scope.go:122] "RemoveContainer" containerID="a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179" Nov 03 20:31:18.167708 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:31:18.167691 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179\": container with ID starting with a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179 not found: ID does not exist" containerID="a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179" Nov 03 20:31:18.167784 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.167715 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179"} err="failed to get container status \"a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179\": rpc error: code = NotFound desc = could not find container \"a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179\": container with ID starting with a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179 not found: ID does not exist" Nov 03 20:31:18.167784 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.167731 4493 scope.go:122] "RemoveContainer" containerID="58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d" Nov 03 20:31:18.167990 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:31:18.167974 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d\": container with ID starting with 58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d not found: ID does not exist" containerID="58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d" Nov 03 20:31:18.168034 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.167995 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d"} err="failed to get container status \"58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d\": rpc error: code = NotFound desc = could not find container \"58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d\": container with ID starting with 58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d not found: ID does not exist" Nov 03 20:31:18.168034 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.168012 4493 scope.go:122] "RemoveContainer" containerID="9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376" Nov 03 20:31:18.168274 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:31:18.168256 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376\": container with ID starting with 9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376 not found: ID does not exist" containerID="9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376" Nov 03 20:31:18.168317 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.168282 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376"} err="failed to get container status \"9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376\": rpc error: code = NotFound desc = could not find container \"9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376\": container with ID starting with 9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376 not found: ID does not exist" Nov 03 20:31:18.168317 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.168297 4493 scope.go:122] "RemoveContainer" containerID="066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea" Nov 03 20:31:18.168577 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:31:18.168556 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea\": container with ID starting with 066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea not found: ID does not exist" containerID="066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea" Nov 03 20:31:18.168653 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.168581 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea"} err="failed to get container status \"066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea\": rpc error: code = NotFound desc = could not find container \"066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea\": container with ID starting with 066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea not found: ID does not exist" Nov 03 20:31:18.168653 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.168596 4493 scope.go:122] "RemoveContainer" containerID="fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2" Nov 03 20:31:18.168887 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:31:18.168866 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2\": container with ID starting with fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2 not found: ID does not exist" containerID="fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2" Nov 03 20:31:18.168924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.168895 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2"} err="failed to get container status \"fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2\": rpc error: code = NotFound desc = could not find container \"fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2\": container with ID starting with fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2 not found: ID does not exist" Nov 03 20:31:18.168924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.168912 4493 scope.go:122] "RemoveContainer" containerID="da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40" Nov 03 20:31:18.169164 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:31:18.169148 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40\": container with ID starting with da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40 not found: ID does not exist" containerID="da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40" Nov 03 20:31:18.169198 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.169169 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40"} err="failed to get container status \"da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40\": rpc error: code = NotFound desc = could not find container \"da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40\": container with ID starting with da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40 not found: ID does not exist" Nov 03 20:31:18.169198 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.169185 4493 scope.go:122] "RemoveContainer" containerID="f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b" Nov 03 20:31:18.169456 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.169439 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b"} err="failed to get container status \"f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b\": rpc error: code = NotFound desc = could not find container \"f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b\": container with ID starting with f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b not found: ID does not exist" Nov 03 20:31:18.169503 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.169458 4493 scope.go:122] "RemoveContainer" containerID="a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179" Nov 03 20:31:18.169666 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.169650 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179"} err="failed to get container status \"a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179\": rpc error: code = NotFound desc = could not find container \"a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179\": container with ID starting with a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179 not found: ID does not exist" Nov 03 20:31:18.169705 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.169667 4493 scope.go:122] "RemoveContainer" containerID="58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d" Nov 03 20:31:18.169866 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.169849 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d"} err="failed to get container status \"58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d\": rpc error: code = NotFound desc = could not find container \"58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d\": container with ID starting with 58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d not found: ID does not exist" Nov 03 20:31:18.169907 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.169866 4493 scope.go:122] "RemoveContainer" containerID="9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376" Nov 03 20:31:18.170098 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.170080 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376"} err="failed to get container status \"9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376\": rpc error: code = NotFound desc = could not find container \"9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376\": container with ID starting with 9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376 not found: ID does not exist" Nov 03 20:31:18.170156 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.170098 4493 scope.go:122] "RemoveContainer" containerID="066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea" Nov 03 20:31:18.170326 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.170293 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea"} err="failed to get container status \"066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea\": rpc error: code = NotFound desc = could not find container \"066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea\": container with ID starting with 066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea not found: ID does not exist" Nov 03 20:31:18.170326 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.170326 4493 scope.go:122] "RemoveContainer" containerID="fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2" Nov 03 20:31:18.170544 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.170524 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2"} err="failed to get container status \"fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2\": rpc error: code = NotFound desc = could not find container \"fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2\": container with ID starting with fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2 not found: ID does not exist" Nov 03 20:31:18.170609 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.170544 4493 scope.go:122] "RemoveContainer" containerID="da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40" Nov 03 20:31:18.170756 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.170733 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40"} err="failed to get container status \"da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40\": rpc error: code = NotFound desc = could not find container \"da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40\": container with ID starting with da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40 not found: ID does not exist" Nov 03 20:31:18.170808 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.170755 4493 scope.go:122] "RemoveContainer" containerID="f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b" Nov 03 20:31:18.170958 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.170939 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b"} err="failed to get container status \"f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b\": rpc error: code = NotFound desc = could not find container \"f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b\": container with ID starting with f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b not found: ID does not exist" Nov 03 20:31:18.170958 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.170958 4493 scope.go:122] "RemoveContainer" containerID="a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179" Nov 03 20:31:18.171156 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.171135 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179"} err="failed to get container status \"a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179\": rpc error: code = NotFound desc = could not find container \"a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179\": container with ID starting with a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179 not found: ID does not exist" Nov 03 20:31:18.171210 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.171156 4493 scope.go:122] "RemoveContainer" containerID="58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d" Nov 03 20:31:18.171379 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.171361 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d"} err="failed to get container status \"58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d\": rpc error: code = NotFound desc = could not find container \"58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d\": container with ID starting with 58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d not found: ID does not exist" Nov 03 20:31:18.171442 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.171379 4493 scope.go:122] "RemoveContainer" containerID="9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376" Nov 03 20:31:18.171588 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.171570 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376"} err="failed to get container status \"9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376\": rpc error: code = NotFound desc = could not find container \"9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376\": container with ID starting with 9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376 not found: ID does not exist" Nov 03 20:31:18.171588 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.171588 4493 scope.go:122] "RemoveContainer" containerID="066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea" Nov 03 20:31:18.171792 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.171772 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea"} err="failed to get container status \"066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea\": rpc error: code = NotFound desc = could not find container \"066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea\": container with ID starting with 066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea not found: ID does not exist" Nov 03 20:31:18.171847 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.171793 4493 scope.go:122] "RemoveContainer" containerID="fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2" Nov 03 20:31:18.171995 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.171978 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2"} err="failed to get container status \"fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2\": rpc error: code = NotFound desc = could not find container \"fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2\": container with ID starting with fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2 not found: ID does not exist" Nov 03 20:31:18.172048 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.171995 4493 scope.go:122] "RemoveContainer" containerID="da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40" Nov 03 20:31:18.172193 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.172173 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40"} err="failed to get container status \"da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40\": rpc error: code = NotFound desc = could not find container \"da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40\": container with ID starting with da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40 not found: ID does not exist" Nov 03 20:31:18.172193 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.172192 4493 scope.go:122] "RemoveContainer" containerID="f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b" Nov 03 20:31:18.172411 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.172396 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b"} err="failed to get container status \"f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b\": rpc error: code = NotFound desc = could not find container \"f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b\": container with ID starting with f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b not found: ID does not exist" Nov 03 20:31:18.172411 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.172411 4493 scope.go:122] "RemoveContainer" containerID="a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179" Nov 03 20:31:18.172644 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.172629 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179"} err="failed to get container status \"a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179\": rpc error: code = NotFound desc = could not find container \"a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179\": container with ID starting with a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179 not found: ID does not exist" Nov 03 20:31:18.172685 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.172644 4493 scope.go:122] "RemoveContainer" containerID="58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d" Nov 03 20:31:18.172879 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.172862 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d"} err="failed to get container status \"58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d\": rpc error: code = NotFound desc = could not find container \"58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d\": container with ID starting with 58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d not found: ID does not exist" Nov 03 20:31:18.172919 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.172879 4493 scope.go:122] "RemoveContainer" containerID="9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376" Nov 03 20:31:18.173082 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.173062 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376"} err="failed to get container status \"9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376\": rpc error: code = NotFound desc = could not find container \"9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376\": container with ID starting with 9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376 not found: ID does not exist" Nov 03 20:31:18.173082 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.173081 4493 scope.go:122] "RemoveContainer" containerID="066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea" Nov 03 20:31:18.173298 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.173283 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea"} err="failed to get container status \"066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea\": rpc error: code = NotFound desc = could not find container \"066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea\": container with ID starting with 066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea not found: ID does not exist" Nov 03 20:31:18.173357 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.173297 4493 scope.go:122] "RemoveContainer" containerID="fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2" Nov 03 20:31:18.173502 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.173487 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2"} err="failed to get container status \"fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2\": rpc error: code = NotFound desc = could not find container \"fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2\": container with ID starting with fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2 not found: ID does not exist" Nov 03 20:31:18.173586 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.173502 4493 scope.go:122] "RemoveContainer" containerID="da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40" Nov 03 20:31:18.173724 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.173709 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40"} err="failed to get container status \"da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40\": rpc error: code = NotFound desc = could not find container \"da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40\": container with ID starting with da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40 not found: ID does not exist" Nov 03 20:31:18.173771 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.173724 4493 scope.go:122] "RemoveContainer" containerID="f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b" Nov 03 20:31:18.173926 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.173907 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b"} err="failed to get container status \"f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b\": rpc error: code = NotFound desc = could not find container \"f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b\": container with ID starting with f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b not found: ID does not exist" Nov 03 20:31:18.173993 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.173928 4493 scope.go:122] "RemoveContainer" containerID="a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179" Nov 03 20:31:18.174146 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.174130 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179"} err="failed to get container status \"a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179\": rpc error: code = NotFound desc = could not find container \"a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179\": container with ID starting with a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179 not found: ID does not exist" Nov 03 20:31:18.174191 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.174147 4493 scope.go:122] "RemoveContainer" containerID="58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d" Nov 03 20:31:18.174348 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.174319 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d"} err="failed to get container status \"58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d\": rpc error: code = NotFound desc = could not find container \"58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d\": container with ID starting with 58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d not found: ID does not exist" Nov 03 20:31:18.174401 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.174348 4493 scope.go:122] "RemoveContainer" containerID="9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376" Nov 03 20:31:18.174514 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.174500 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376"} err="failed to get container status \"9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376\": rpc error: code = NotFound desc = could not find container \"9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376\": container with ID starting with 9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376 not found: ID does not exist" Nov 03 20:31:18.174555 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.174513 4493 scope.go:122] "RemoveContainer" containerID="066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea" Nov 03 20:31:18.174716 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.174698 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea"} err="failed to get container status \"066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea\": rpc error: code = NotFound desc = could not find container \"066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea\": container with ID starting with 066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea not found: ID does not exist" Nov 03 20:31:18.174786 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.174717 4493 scope.go:122] "RemoveContainer" containerID="fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2" Nov 03 20:31:18.174940 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.174919 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2"} err="failed to get container status \"fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2\": rpc error: code = NotFound desc = could not find container \"fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2\": container with ID starting with fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2 not found: ID does not exist" Nov 03 20:31:18.174940 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.174940 4493 scope.go:122] "RemoveContainer" containerID="da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40" Nov 03 20:31:18.175178 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.175158 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40"} err="failed to get container status \"da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40\": rpc error: code = NotFound desc = could not find container \"da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40\": container with ID starting with da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40 not found: ID does not exist" Nov 03 20:31:18.175256 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.175180 4493 scope.go:122] "RemoveContainer" containerID="f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b" Nov 03 20:31:18.175409 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.175393 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b"} err="failed to get container status \"f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b\": rpc error: code = NotFound desc = could not find container \"f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b\": container with ID starting with f0acfa7dc92517e58ecb344e92997dcb4b0ef1f17bc8b5714e48fc817ec5800b not found: ID does not exist" Nov 03 20:31:18.175482 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.175411 4493 scope.go:122] "RemoveContainer" containerID="a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179" Nov 03 20:31:18.175633 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.175615 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179"} err="failed to get container status \"a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179\": rpc error: code = NotFound desc = could not find container \"a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179\": container with ID starting with a53d5994df849bc70514c41cf2ab41e8c3e8774a797c7cbd37d5572cb8a20179 not found: ID does not exist" Nov 03 20:31:18.175686 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.175634 4493 scope.go:122] "RemoveContainer" containerID="58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d" Nov 03 20:31:18.175876 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.175858 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d"} err="failed to get container status \"58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d\": rpc error: code = NotFound desc = could not find container \"58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d\": container with ID starting with 58ea93867b4040de046d29c875e73835a224849faebf33af91a9fbfd45fb850d not found: ID does not exist" Nov 03 20:31:18.175944 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.175877 4493 scope.go:122] "RemoveContainer" containerID="9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376" Nov 03 20:31:18.176086 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.176067 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376"} err="failed to get container status \"9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376\": rpc error: code = NotFound desc = could not find container \"9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376\": container with ID starting with 9f62846c14657e44c4d465895efff2d9dadaa562c8cc43c70aafaf678302b376 not found: ID does not exist" Nov 03 20:31:18.176131 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.176087 4493 scope.go:122] "RemoveContainer" containerID="066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea" Nov 03 20:31:18.176272 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.176255 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea"} err="failed to get container status \"066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea\": rpc error: code = NotFound desc = could not find container \"066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea\": container with ID starting with 066a1af0a379d521ebcf9c38ead361759553bbd2275ec62b37a2f0729cde15ea not found: ID does not exist" Nov 03 20:31:18.176358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.176272 4493 scope.go:122] "RemoveContainer" containerID="fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2" Nov 03 20:31:18.176490 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.176472 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2"} err="failed to get container status \"fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2\": rpc error: code = NotFound desc = could not find container \"fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2\": container with ID starting with fbdae9aee68e45db864a92a2673b052d24419db3d90d4098215b6b20a4a6b5d2 not found: ID does not exist" Nov 03 20:31:18.176552 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.176490 4493 scope.go:122] "RemoveContainer" containerID="da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40" Nov 03 20:31:18.176701 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.176684 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40"} err="failed to get container status \"da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40\": rpc error: code = NotFound desc = could not find container \"da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40\": container with ID starting with da73e9d466f2bdb97308b1f68d94a32261d74598fdf80484fe886621ab60dc40 not found: ID does not exist" Nov 03 20:31:18.275310 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.275285 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:31:18.280094 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.280077 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:31:18.304430 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.304394 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:31:18.304701 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.304685 4493 cpu_manager.go:414] "containerMap: removing container" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerName="kube-rbac-proxy-web" Nov 03 20:31:18.304769 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.304704 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerName="kube-rbac-proxy-web" Nov 03 20:31:18.304769 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.304729 4493 cpu_manager.go:414] "containerMap: removing container" podUID="2296e3c6-6709-4fb0-8384-b0800d874d32" containerName="registry" Nov 03 20:31:18.304769 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.304738 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2296e3c6-6709-4fb0-8384-b0800d874d32" containerName="registry" Nov 03 20:31:18.304769 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.304748 4493 cpu_manager.go:414] "containerMap: removing container" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerName="init-config-reloader" Nov 03 20:31:18.304769 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.304755 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerName="init-config-reloader" Nov 03 20:31:18.304769 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.304766 4493 cpu_manager.go:414] "containerMap: removing container" podUID="08c9d7c8-dbd8-46ad-92f4-2a009e83abc3" containerName="insights-gathering" Nov 03 20:31:18.305060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.304774 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c9d7c8-dbd8-46ad-92f4-2a009e83abc3" containerName="insights-gathering" Nov 03 20:31:18.305060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.304785 4493 cpu_manager.go:414] "containerMap: removing container" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerName="config-reloader" Nov 03 20:31:18.305060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.304793 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerName="config-reloader" Nov 03 20:31:18.305060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.304814 4493 cpu_manager.go:414] "containerMap: removing container" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerName="prometheus" Nov 03 20:31:18.305060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.304821 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerName="prometheus" Nov 03 20:31:18.305060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.304832 4493 cpu_manager.go:414] "containerMap: removing container" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerName="kube-rbac-proxy-thanos" Nov 03 20:31:18.305060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.304839 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerName="kube-rbac-proxy-thanos" Nov 03 20:31:18.305060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.304861 4493 cpu_manager.go:414] "containerMap: removing container" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerName="thanos-sidecar" Nov 03 20:31:18.305060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.304868 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerName="thanos-sidecar" Nov 03 20:31:18.305060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.304877 4493 cpu_manager.go:414] "containerMap: removing container" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerName="kube-rbac-proxy" Nov 03 20:31:18.305060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.304885 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerName="kube-rbac-proxy" Nov 03 20:31:18.305060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.304946 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerName="thanos-sidecar" Nov 03 20:31:18.305060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.304957 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerName="kube-rbac-proxy" Nov 03 20:31:18.305060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.304967 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerName="kube-rbac-proxy-web" Nov 03 20:31:18.305060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.304978 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="2296e3c6-6709-4fb0-8384-b0800d874d32" containerName="registry" Nov 03 20:31:18.305060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.304986 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c9d7c8-dbd8-46ad-92f4-2a009e83abc3" containerName="insights-gathering" Nov 03 20:31:18.305060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.304995 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerName="config-reloader" Nov 03 20:31:18.305060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.305004 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerName="prometheus" Nov 03 20:31:18.305060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.305011 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" containerName="kube-rbac-proxy-thanos" Nov 03 20:31:18.309649 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.309631 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.359847 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.359813 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 20:31:18.359976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.359878 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 20:31:18.359976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.359962 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 20:31:18.360133 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.360117 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 20:31:18.360206 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.360192 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 20:31:18.360262 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.360214 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 20:31:18.360262 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.360236 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 20:31:18.360262 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.360241 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 20:31:18.360638 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.360292 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 20:31:18.360638 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.360422 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e4gqi9de47aks\"" Nov 03 20:31:18.360638 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.360427 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 20:31:18.360638 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.360559 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jrwnj\"" Nov 03 20:31:18.361003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.360985 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 20:31:18.407472 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.407406 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 20:31:18.407472 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.407408 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 20:31:18.407472 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.407445 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 20:31:18.407640 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.407525 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 20:31:18.425586 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.425560 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.425752 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.425656 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.425752 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.425704 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.426110 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.425792 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.426110 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.426001 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.426110 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.426025 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.426110 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.426055 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-config-out\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.426264 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.426112 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.426264 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.426152 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.426264 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.426222 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-config\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.426264 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.426242 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.426264 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.426257 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.426464 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.426277 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.426464 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.426368 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h95m5\" (UniqueName: \"kubernetes.io/projected/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-kube-api-access-h95m5\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.426464 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.426395 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.426464 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.426422 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.426464 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.426458 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.426615 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.426521 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.426647 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.426623 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.426679 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.426649 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-web-config\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.511062 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.511030 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:31:18.527930 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.527907 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.528070 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.527936 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.528070 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.527956 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.528070 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.527971 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.528232 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.528114 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.528232 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.528157 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-config-out\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.528232 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.528190 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.528232 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.528220 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.528445 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.528289 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-config\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.528445 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.528314 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.528445 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.528355 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.528445 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.528388 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.528445 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.528430 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h95m5\" (UniqueName: \"kubernetes.io/projected/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-kube-api-access-h95m5\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.528689 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.528454 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.528689 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.528482 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.528689 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.528510 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.528689 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.528540 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.528689 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.528587 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.528689 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.528611 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-web-config\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.528689 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.528651 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.529024 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.528736 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.529024 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.528924 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.529914 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.529887 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.530785 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.530756 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.533298 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.532955 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.533298 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.533193 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.533298 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.533239 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.533537 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.533496 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.533592 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.533558 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.534317 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.534291 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.534437 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.534380 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.534895 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.534813 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-config-out\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.534895 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.534813 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.535039 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.534995 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.536088 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.536049 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-web-config\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.536358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.536311 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.536658 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.536639 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-config\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.536945 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.536921 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.538581 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.538557 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.561047 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.560146 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ea6ce6d-cda8-4676-a237-01e667f6815a" path="/var/lib/kubelet/pods/6ea6ce6d-cda8-4676-a237-01e667f6815a/volumes" Nov 03 20:31:18.582496 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.582447 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h95m5\" (UniqueName: \"kubernetes.io/projected/5a03d57d-47f6-4ae1-9e1a-62cbe72d453b-kube-api-access-h95m5\") pod \"prometheus-k8s-0\" (UID: \"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:18.628353 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:18.628300 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:19.062823 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:19.061810 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:31:19.067183 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:31:19.067155 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a03d57d_47f6_4ae1_9e1a_62cbe72d453b.slice/crio-ce0a3c112a0d962f1e0e601b2eb388d1ff8d3e66471b08d400faf71ae50b63bf WatchSource:0}: Error finding container ce0a3c112a0d962f1e0e601b2eb388d1ff8d3e66471b08d400faf71ae50b63bf: Status 404 returned error can't find the container with id ce0a3c112a0d962f1e0e601b2eb388d1ff8d3e66471b08d400faf71ae50b63bf Nov 03 20:31:19.123861 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:19.123831 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b","Type":"ContainerStarted","Data":"ce0a3c112a0d962f1e0e601b2eb388d1ff8d3e66471b08d400faf71ae50b63bf"} Nov 03 20:31:20.129603 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:20.129570 4493 generic.go:356] "Generic (PLEG): container finished" podID="5a03d57d-47f6-4ae1-9e1a-62cbe72d453b" containerID="e60e9298fed96ab07174d9a2b063240ebae533b2c87c968ec06d61471996c304" exitCode=0 Nov 03 20:31:20.129996 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:20.129653 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b","Type":"ContainerDied","Data":"e60e9298fed96ab07174d9a2b063240ebae533b2c87c968ec06d61471996c304"} Nov 03 20:31:21.137158 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:21.137123 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b","Type":"ContainerStarted","Data":"4a117fa5dbdfb0e308db06fbf3c55a760db8994e35c63d5504d7c90ffd01826a"} Nov 03 20:31:21.137695 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:21.137203 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b","Type":"ContainerStarted","Data":"9106d414e7154efcd5832d9f6d32da795b73bcb0e079a2f313520631e2944411"} Nov 03 20:31:21.137695 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:21.137244 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b","Type":"ContainerStarted","Data":"4fdcf3cd87f0d95af4d092b98cb7fe7175c5cdfe3b0a0636da8b083582d9bba6"} Nov 03 20:31:21.137695 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:21.137285 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b","Type":"ContainerStarted","Data":"12dea09734478a4a9e5c1f6ce02c8c25902da41141441a0dff192b9b7f0b04ec"} Nov 03 20:31:21.137695 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:21.137323 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b","Type":"ContainerStarted","Data":"39e3a591ab397ebdf70a45b7d99a981f52138641dc0e2a55e6b119ea58be8236"} Nov 03 20:31:21.137695 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:21.137379 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a03d57d-47f6-4ae1-9e1a-62cbe72d453b","Type":"ContainerStarted","Data":"fedf878aa94f1a4602642f208bf3d8e2745ff9096031751fb91c1048579d8d64"} Nov 03 20:31:21.255309 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:21.255263 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.239964531 podStartE2EDuration="3.255249776s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.285245ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:21.253966015 +0000 UTC m=+152.811956564" watchObservedRunningTime="2026-11-03 20:31:21.255249776 +0000 UTC m=+152.813240275" Nov 03 20:31:21.742452 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:31:21.742416 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:31:23.628433 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:31:23.628395 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:31:51.742572 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:31:51.742526 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:32:18.629367 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:32:18.629312 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:32:18.645353 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:32:18.645311 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:32:19.353236 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:32:19.353211 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:32:21.742077 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:32:21.742042 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:32:51.742689 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:32:51.742650 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:33:21.742420 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:33:21.742382 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:33:48.562711 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:33:48.562686 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 20:33:48.563446 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:33:48.563426 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 20:33:51.742804 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:33:51.742770 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:34:21.742600 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:34:21.742568 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:34:51.742799 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:34:51.742757 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:35:01.153692 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.153657 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb"] Nov 03 20:35:01.158691 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.158665 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb" Nov 03 20:35:01.177844 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.177820 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c"] Nov 03 20:35:01.181045 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.181029 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c" Nov 03 20:35:01.202347 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.202277 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx"] Nov 03 20:35:01.205345 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.205317 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx" Nov 03 20:35:01.224046 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.224024 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-2hhmp\"" Nov 03 20:35:01.225172 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.225151 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb"] Nov 03 20:35:01.248273 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.248254 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 20:35:01.248370 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.248282 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 20:35:01.259854 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.259833 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx"] Nov 03 20:35:01.268647 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.268627 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c"] Nov 03 20:35:01.273192 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.273175 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4081add1-cd6a-48a0-aaa0-d76790121ecb-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx\" (UID: \"4081add1-cd6a-48a0-aaa0-d76790121ecb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx" Nov 03 20:35:01.273282 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.273208 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/067cdd51-58e6-4823-a3ec-4e03ea366c58-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb\" (UID: \"067cdd51-58e6-4823-a3ec-4e03ea366c58\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb" Nov 03 20:35:01.273282 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.273235 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scs54\" (UniqueName: \"kubernetes.io/projected/067cdd51-58e6-4823-a3ec-4e03ea366c58-kube-api-access-scs54\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb\" (UID: \"067cdd51-58e6-4823-a3ec-4e03ea366c58\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb" Nov 03 20:35:01.273394 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.273292 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae9e9c00-2e1d-472a-b037-dd0ce0f229fd-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c\" (UID: \"ae9e9c00-2e1d-472a-b037-dd0ce0f229fd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c" Nov 03 20:35:01.273474 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.273459 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll846\" (UniqueName: \"kubernetes.io/projected/4081add1-cd6a-48a0-aaa0-d76790121ecb-kube-api-access-ll846\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx\" (UID: \"4081add1-cd6a-48a0-aaa0-d76790121ecb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx" Nov 03 20:35:01.273508 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.273495 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/067cdd51-58e6-4823-a3ec-4e03ea366c58-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb\" (UID: \"067cdd51-58e6-4823-a3ec-4e03ea366c58\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb" Nov 03 20:35:01.273577 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.273555 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae9e9c00-2e1d-472a-b037-dd0ce0f229fd-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c\" (UID: \"ae9e9c00-2e1d-472a-b037-dd0ce0f229fd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c" Nov 03 20:35:01.273623 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.273609 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4081add1-cd6a-48a0-aaa0-d76790121ecb-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx\" (UID: \"4081add1-cd6a-48a0-aaa0-d76790121ecb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx" Nov 03 20:35:01.273658 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.273642 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f8ph\" (UniqueName: \"kubernetes.io/projected/ae9e9c00-2e1d-472a-b037-dd0ce0f229fd-kube-api-access-5f8ph\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c\" (UID: \"ae9e9c00-2e1d-472a-b037-dd0ce0f229fd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c" Nov 03 20:35:01.310224 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.310205 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-whd2z\"" Nov 03 20:35:01.310303 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.310258 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-zqvl9\"" Nov 03 20:35:01.374759 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.374732 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/067cdd51-58e6-4823-a3ec-4e03ea366c58-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb\" (UID: \"067cdd51-58e6-4823-a3ec-4e03ea366c58\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb" Nov 03 20:35:01.374900 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.374772 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scs54\" (UniqueName: \"kubernetes.io/projected/067cdd51-58e6-4823-a3ec-4e03ea366c58-kube-api-access-scs54\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb\" (UID: \"067cdd51-58e6-4823-a3ec-4e03ea366c58\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb" Nov 03 20:35:01.374900 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.374793 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae9e9c00-2e1d-472a-b037-dd0ce0f229fd-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c\" (UID: \"ae9e9c00-2e1d-472a-b037-dd0ce0f229fd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c" Nov 03 20:35:01.374900 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.374813 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll846\" (UniqueName: \"kubernetes.io/projected/4081add1-cd6a-48a0-aaa0-d76790121ecb-kube-api-access-ll846\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx\" (UID: \"4081add1-cd6a-48a0-aaa0-d76790121ecb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx" Nov 03 20:35:01.374900 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.374847 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/067cdd51-58e6-4823-a3ec-4e03ea366c58-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb\" (UID: \"067cdd51-58e6-4823-a3ec-4e03ea366c58\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb" Nov 03 20:35:01.374900 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.374868 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae9e9c00-2e1d-472a-b037-dd0ce0f229fd-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c\" (UID: \"ae9e9c00-2e1d-472a-b037-dd0ce0f229fd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c" Nov 03 20:35:01.374900 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.374895 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4081add1-cd6a-48a0-aaa0-d76790121ecb-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx\" (UID: \"4081add1-cd6a-48a0-aaa0-d76790121ecb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx" Nov 03 20:35:01.375201 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.374924 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f8ph\" (UniqueName: \"kubernetes.io/projected/ae9e9c00-2e1d-472a-b037-dd0ce0f229fd-kube-api-access-5f8ph\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c\" (UID: \"ae9e9c00-2e1d-472a-b037-dd0ce0f229fd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c" Nov 03 20:35:01.375201 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.374961 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4081add1-cd6a-48a0-aaa0-d76790121ecb-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx\" (UID: \"4081add1-cd6a-48a0-aaa0-d76790121ecb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx" Nov 03 20:35:01.375299 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.375226 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/067cdd51-58e6-4823-a3ec-4e03ea366c58-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb\" (UID: \"067cdd51-58e6-4823-a3ec-4e03ea366c58\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb" Nov 03 20:35:01.375375 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.375293 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae9e9c00-2e1d-472a-b037-dd0ce0f229fd-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c\" (UID: \"ae9e9c00-2e1d-472a-b037-dd0ce0f229fd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c" Nov 03 20:35:01.375375 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.375226 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae9e9c00-2e1d-472a-b037-dd0ce0f229fd-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c\" (UID: \"ae9e9c00-2e1d-472a-b037-dd0ce0f229fd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c" Nov 03 20:35:01.375450 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.375379 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4081add1-cd6a-48a0-aaa0-d76790121ecb-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx\" (UID: \"4081add1-cd6a-48a0-aaa0-d76790121ecb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx" Nov 03 20:35:01.375450 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.375400 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4081add1-cd6a-48a0-aaa0-d76790121ecb-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx\" (UID: \"4081add1-cd6a-48a0-aaa0-d76790121ecb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx" Nov 03 20:35:01.375450 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.375384 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/067cdd51-58e6-4823-a3ec-4e03ea366c58-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb\" (UID: \"067cdd51-58e6-4823-a3ec-4e03ea366c58\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb" Nov 03 20:35:01.430236 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.430209 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scs54\" (UniqueName: \"kubernetes.io/projected/067cdd51-58e6-4823-a3ec-4e03ea366c58-kube-api-access-scs54\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb\" (UID: \"067cdd51-58e6-4823-a3ec-4e03ea366c58\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb" Nov 03 20:35:01.430483 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.430462 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f8ph\" (UniqueName: \"kubernetes.io/projected/ae9e9c00-2e1d-472a-b037-dd0ce0f229fd-kube-api-access-5f8ph\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c\" (UID: \"ae9e9c00-2e1d-472a-b037-dd0ce0f229fd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c" Nov 03 20:35:01.430535 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.430462 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll846\" (UniqueName: \"kubernetes.io/projected/4081add1-cd6a-48a0-aaa0-d76790121ecb-kube-api-access-ll846\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx\" (UID: \"4081add1-cd6a-48a0-aaa0-d76790121ecb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx" Nov 03 20:35:01.473678 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.473630 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb" Nov 03 20:35:01.495999 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.495971 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c" Nov 03 20:35:01.523957 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.523698 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx" Nov 03 20:35:01.872482 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.872451 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb"] Nov 03 20:35:01.878683 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:35:01.878661 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067cdd51_58e6_4823_a3ec_4e03ea366c58.slice/crio-25ae0543b35722d4be028f693c05f6c9be6bc1b96c76dff2d891e23e415bfd60 WatchSource:0}: Error finding container 25ae0543b35722d4be028f693c05f6c9be6bc1b96c76dff2d891e23e415bfd60: Status 404 returned error can't find the container with id 25ae0543b35722d4be028f693c05f6c9be6bc1b96c76dff2d891e23e415bfd60 Nov 03 20:35:01.880473 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.880456 4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:35:01.896381 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.896358 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c"] Nov 03 20:35:01.902539 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:35:01.902512 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9e9c00_2e1d_472a_b037_dd0ce0f229fd.slice/crio-0754ced8a53a23d04fd31b4c59ba7479bc30daaf46da4b9eb73228d50bb01374 WatchSource:0}: Error finding container 0754ced8a53a23d04fd31b4c59ba7479bc30daaf46da4b9eb73228d50bb01374: Status 404 returned error can't find the container with id 0754ced8a53a23d04fd31b4c59ba7479bc30daaf46da4b9eb73228d50bb01374 Nov 03 20:35:01.925257 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:01.925228 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx"] Nov 03 20:35:01.932091 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:35:01.932067 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4081add1_cd6a_48a0_aaa0_d76790121ecb.slice/crio-2cbafc7ad9ec4a2c8e4818a1122da84a9673432c2938e48a5df9d89fc4d162e7 WatchSource:0}: Error finding container 2cbafc7ad9ec4a2c8e4818a1122da84a9673432c2938e48a5df9d89fc4d162e7: Status 404 returned error can't find the container with id 2cbafc7ad9ec4a2c8e4818a1122da84a9673432c2938e48a5df9d89fc4d162e7 Nov 03 20:35:02.878943 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:02.878905 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c" event={"ID":"ae9e9c00-2e1d-472a-b037-dd0ce0f229fd","Type":"ContainerStarted","Data":"0754ced8a53a23d04fd31b4c59ba7479bc30daaf46da4b9eb73228d50bb01374"} Nov 03 20:35:02.880031 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:02.880010 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb" event={"ID":"067cdd51-58e6-4823-a3ec-4e03ea366c58","Type":"ContainerStarted","Data":"25ae0543b35722d4be028f693c05f6c9be6bc1b96c76dff2d891e23e415bfd60"} Nov 03 20:35:02.880973 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:02.880952 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx" event={"ID":"4081add1-cd6a-48a0-aaa0-d76790121ecb","Type":"ContainerStarted","Data":"2cbafc7ad9ec4a2c8e4818a1122da84a9673432c2938e48a5df9d89fc4d162e7"} Nov 03 20:35:09.915964 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:09.915931 4493 generic.go:356] "Generic (PLEG): container finished" podID="4081add1-cd6a-48a0-aaa0-d76790121ecb" containerID="6c107e7ae3ec4417655cdb247875b784193041960e8858cd69d53683ae994e30" exitCode=0 Nov 03 20:35:09.916306 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:09.915967 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx" event={"ID":"4081add1-cd6a-48a0-aaa0-d76790121ecb","Type":"ContainerDied","Data":"6c107e7ae3ec4417655cdb247875b784193041960e8858cd69d53683ae994e30"} Nov 03 20:35:10.920805 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:10.920773 4493 generic.go:356] "Generic (PLEG): container finished" podID="ae9e9c00-2e1d-472a-b037-dd0ce0f229fd" containerID="f4b230d5a2e93345c4fd27bd2fe960ee6e3311240acd38f43d9ba488609fa2e6" exitCode=0 Nov 03 20:35:10.921263 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:10.920842 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c" event={"ID":"ae9e9c00-2e1d-472a-b037-dd0ce0f229fd","Type":"ContainerDied","Data":"f4b230d5a2e93345c4fd27bd2fe960ee6e3311240acd38f43d9ba488609fa2e6"} Nov 03 20:35:10.922455 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:10.922430 4493 generic.go:356] "Generic (PLEG): container finished" podID="067cdd51-58e6-4823-a3ec-4e03ea366c58" containerID="79d14583937caa357e356a81b6a68cac8610943180ac96190b837e57b8b0eedb" exitCode=0 Nov 03 20:35:10.922530 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:10.922510 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb" event={"ID":"067cdd51-58e6-4823-a3ec-4e03ea366c58","Type":"ContainerDied","Data":"79d14583937caa357e356a81b6a68cac8610943180ac96190b837e57b8b0eedb"} Nov 03 20:35:11.928185 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:11.928149 4493 generic.go:356] "Generic (PLEG): container finished" podID="4081add1-cd6a-48a0-aaa0-d76790121ecb" containerID="5cebfe16640d20f9721b1e44616318203d7c3366514f299b0ebd92cfcd17a77f" exitCode=0 Nov 03 20:35:11.928607 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:11.928228 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx" event={"ID":"4081add1-cd6a-48a0-aaa0-d76790121ecb","Type":"ContainerDied","Data":"5cebfe16640d20f9721b1e44616318203d7c3366514f299b0ebd92cfcd17a77f"} Nov 03 20:35:12.933834 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:12.933747 4493 generic.go:356] "Generic (PLEG): container finished" podID="ae9e9c00-2e1d-472a-b037-dd0ce0f229fd" containerID="8174603b3107f031131507321329f4cb6d7ce09b44228db8e632f15e587ce866" exitCode=0 Nov 03 20:35:12.933834 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:12.933812 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c" event={"ID":"ae9e9c00-2e1d-472a-b037-dd0ce0f229fd","Type":"ContainerDied","Data":"8174603b3107f031131507321329f4cb6d7ce09b44228db8e632f15e587ce866"} Nov 03 20:35:12.935777 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:12.935749 4493 generic.go:356] "Generic (PLEG): container finished" podID="067cdd51-58e6-4823-a3ec-4e03ea366c58" containerID="777435cbbd46b596576c02033e52c1385c3c69118bcf7e40d21d2206ec874859" exitCode=0 Nov 03 20:35:12.935869 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:12.935847 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb" event={"ID":"067cdd51-58e6-4823-a3ec-4e03ea366c58","Type":"ContainerDied","Data":"777435cbbd46b596576c02033e52c1385c3c69118bcf7e40d21d2206ec874859"} Nov 03 20:35:20.980639 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:20.980469 4493 generic.go:356] "Generic (PLEG): container finished" podID="4081add1-cd6a-48a0-aaa0-d76790121ecb" containerID="bbe36747c6d38f3014915af9b7ffa8139f6e2271bcaeb1b118d886338807d353" exitCode=0 Nov 03 20:35:20.980639 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:20.980550 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx" event={"ID":"4081add1-cd6a-48a0-aaa0-d76790121ecb","Type":"ContainerDied","Data":"bbe36747c6d38f3014915af9b7ffa8139f6e2271bcaeb1b118d886338807d353"} Nov 03 20:35:21.743456 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:35:21.743417 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:35:21.986282 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:21.986252 4493 generic.go:356] "Generic (PLEG): container finished" podID="ae9e9c00-2e1d-472a-b037-dd0ce0f229fd" containerID="39aa8b014edcc76b221c04ed0dfea5eca1734da6ed6f4188baa67225e55150b3" exitCode=0 Nov 03 20:35:21.986713 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:21.986321 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c" event={"ID":"ae9e9c00-2e1d-472a-b037-dd0ce0f229fd","Type":"ContainerDied","Data":"39aa8b014edcc76b221c04ed0dfea5eca1734da6ed6f4188baa67225e55150b3"} Nov 03 20:35:21.988453 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:21.988432 4493 generic.go:356] "Generic (PLEG): container finished" podID="067cdd51-58e6-4823-a3ec-4e03ea366c58" containerID="3b6095aac613094cb7398e151e5d4e1f03c51d426f64255e37cd9471db2b609f" exitCode=0 Nov 03 20:35:21.988551 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:21.988519 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb" event={"ID":"067cdd51-58e6-4823-a3ec-4e03ea366c58","Type":"ContainerDied","Data":"3b6095aac613094cb7398e151e5d4e1f03c51d426f64255e37cd9471db2b609f"} Nov 03 20:35:22.108508 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:22.108484 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx" Nov 03 20:35:22.158151 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:22.158118 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4081add1-cd6a-48a0-aaa0-d76790121ecb-kube-api-access-ll846\" (UniqueName: \"kubernetes.io/projected/4081add1-cd6a-48a0-aaa0-d76790121ecb-kube-api-access-ll846\") pod \"4081add1-cd6a-48a0-aaa0-d76790121ecb\" (UID: \"4081add1-cd6a-48a0-aaa0-d76790121ecb\") " Nov 03 20:35:22.158283 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:22.158209 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4081add1-cd6a-48a0-aaa0-d76790121ecb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4081add1-cd6a-48a0-aaa0-d76790121ecb-bundle\") pod \"4081add1-cd6a-48a0-aaa0-d76790121ecb\" (UID: \"4081add1-cd6a-48a0-aaa0-d76790121ecb\") " Nov 03 20:35:22.158672 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:22.158649 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4081add1-cd6a-48a0-aaa0-d76790121ecb-bundle" pod "4081add1-cd6a-48a0-aaa0-d76790121ecb" (UID: "4081add1-cd6a-48a0-aaa0-d76790121ecb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:22.160409 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:22.160386 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4081add1-cd6a-48a0-aaa0-d76790121ecb-kube-api-access-ll846" pod "4081add1-cd6a-48a0-aaa0-d76790121ecb" (UID: "4081add1-cd6a-48a0-aaa0-d76790121ecb"). InnerVolumeSpecName "kube-api-access-ll846". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:22.259549 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:22.259523 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4081add1-cd6a-48a0-aaa0-d76790121ecb-util\" (UniqueName: \"kubernetes.io/empty-dir/4081add1-cd6a-48a0-aaa0-d76790121ecb-util\") pod \"4081add1-cd6a-48a0-aaa0-d76790121ecb\" (UID: \"4081add1-cd6a-48a0-aaa0-d76790121ecb\") " Nov 03 20:35:22.259713 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:22.259636 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll846\" (UniqueName: \"kubernetes.io/projected/4081add1-cd6a-48a0-aaa0-d76790121ecb-kube-api-access-ll846\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:22.259713 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:22.259648 4493 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4081add1-cd6a-48a0-aaa0-d76790121ecb-bundle\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:22.263737 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:22.263713 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4081add1-cd6a-48a0-aaa0-d76790121ecb-util" pod "4081add1-cd6a-48a0-aaa0-d76790121ecb" (UID: "4081add1-cd6a-48a0-aaa0-d76790121ecb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:22.360412 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:22.360382 4493 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4081add1-cd6a-48a0-aaa0-d76790121ecb-util\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:22.993592 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:22.993567 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx" Nov 03 20:35:22.994041 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:22.993605 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gn4vx" event={"ID":"4081add1-cd6a-48a0-aaa0-d76790121ecb","Type":"ContainerDied","Data":"2cbafc7ad9ec4a2c8e4818a1122da84a9673432c2938e48a5df9d89fc4d162e7"} Nov 03 20:35:22.994041 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:22.993646 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cbafc7ad9ec4a2c8e4818a1122da84a9673432c2938e48a5df9d89fc4d162e7" Nov 03 20:35:23.163407 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:23.163380 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c" Nov 03 20:35:23.167930 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:23.167912 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae9e9c00-2e1d-472a-b037-dd0ce0f229fd-kube-api-access-5f8ph\" (UniqueName: \"kubernetes.io/projected/ae9e9c00-2e1d-472a-b037-dd0ce0f229fd-kube-api-access-5f8ph\") pod \"ae9e9c00-2e1d-472a-b037-dd0ce0f229fd\" (UID: \"ae9e9c00-2e1d-472a-b037-dd0ce0f229fd\") " Nov 03 20:35:23.168038 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:23.167964 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae9e9c00-2e1d-472a-b037-dd0ce0f229fd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae9e9c00-2e1d-472a-b037-dd0ce0f229fd-bundle\") pod \"ae9e9c00-2e1d-472a-b037-dd0ce0f229fd\" (UID: \"ae9e9c00-2e1d-472a-b037-dd0ce0f229fd\") " Nov 03 20:35:23.168038 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:23.167993 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae9e9c00-2e1d-472a-b037-dd0ce0f229fd-util\" (UniqueName: \"kubernetes.io/empty-dir/ae9e9c00-2e1d-472a-b037-dd0ce0f229fd-util\") pod \"ae9e9c00-2e1d-472a-b037-dd0ce0f229fd\" (UID: \"ae9e9c00-2e1d-472a-b037-dd0ce0f229fd\") " Nov 03 20:35:23.168598 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:23.168576 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9e9c00-2e1d-472a-b037-dd0ce0f229fd-bundle" pod "ae9e9c00-2e1d-472a-b037-dd0ce0f229fd" (UID: "ae9e9c00-2e1d-472a-b037-dd0ce0f229fd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:23.170078 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:23.170057 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae9e9c00-2e1d-472a-b037-dd0ce0f229fd-kube-api-access-5f8ph" pod "ae9e9c00-2e1d-472a-b037-dd0ce0f229fd" (UID: "ae9e9c00-2e1d-472a-b037-dd0ce0f229fd"). InnerVolumeSpecName "kube-api-access-5f8ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:23.171849 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:23.171828 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9e9c00-2e1d-472a-b037-dd0ce0f229fd-util" pod "ae9e9c00-2e1d-472a-b037-dd0ce0f229fd" (UID: "ae9e9c00-2e1d-472a-b037-dd0ce0f229fd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:23.183346 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:23.183308 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb" Nov 03 20:35:23.269123 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:23.269076 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/067cdd51-58e6-4823-a3ec-4e03ea366c58-bundle\" (UniqueName: \"kubernetes.io/empty-dir/067cdd51-58e6-4823-a3ec-4e03ea366c58-bundle\") pod \"067cdd51-58e6-4823-a3ec-4e03ea366c58\" (UID: \"067cdd51-58e6-4823-a3ec-4e03ea366c58\") " Nov 03 20:35:23.269123 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:23.269118 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/067cdd51-58e6-4823-a3ec-4e03ea366c58-kube-api-access-scs54\" (UniqueName: \"kubernetes.io/projected/067cdd51-58e6-4823-a3ec-4e03ea366c58-kube-api-access-scs54\") pod \"067cdd51-58e6-4823-a3ec-4e03ea366c58\" (UID: \"067cdd51-58e6-4823-a3ec-4e03ea366c58\") " Nov 03 20:35:23.269247 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:23.269165 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/067cdd51-58e6-4823-a3ec-4e03ea366c58-util\" (UniqueName: \"kubernetes.io/empty-dir/067cdd51-58e6-4823-a3ec-4e03ea366c58-util\") pod \"067cdd51-58e6-4823-a3ec-4e03ea366c58\" (UID: \"067cdd51-58e6-4823-a3ec-4e03ea366c58\") " Nov 03 20:35:23.269295 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:23.269283 4493 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae9e9c00-2e1d-472a-b037-dd0ce0f229fd-util\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:23.269368 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:23.269302 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f8ph\" (UniqueName: \"kubernetes.io/projected/ae9e9c00-2e1d-472a-b037-dd0ce0f229fd-kube-api-access-5f8ph\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:23.269368 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:23.269316 4493 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae9e9c00-2e1d-472a-b037-dd0ce0f229fd-bundle\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:23.269640 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:23.269614 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/067cdd51-58e6-4823-a3ec-4e03ea366c58-bundle" pod "067cdd51-58e6-4823-a3ec-4e03ea366c58" (UID: "067cdd51-58e6-4823-a3ec-4e03ea366c58"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:23.271546 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:23.271520 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/067cdd51-58e6-4823-a3ec-4e03ea366c58-kube-api-access-scs54" pod "067cdd51-58e6-4823-a3ec-4e03ea366c58" (UID: "067cdd51-58e6-4823-a3ec-4e03ea366c58"). InnerVolumeSpecName "kube-api-access-scs54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:23.273700 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:23.273672 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/067cdd51-58e6-4823-a3ec-4e03ea366c58-util" pod "067cdd51-58e6-4823-a3ec-4e03ea366c58" (UID: "067cdd51-58e6-4823-a3ec-4e03ea366c58"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:23.370397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:23.370368 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scs54\" (UniqueName: \"kubernetes.io/projected/067cdd51-58e6-4823-a3ec-4e03ea366c58-kube-api-access-scs54\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:23.370397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:23.370392 4493 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/067cdd51-58e6-4823-a3ec-4e03ea366c58-util\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:23.370397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:23.370401 4493 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/067cdd51-58e6-4823-a3ec-4e03ea366c58-bundle\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:23.999525 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:23.999502 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c" Nov 03 20:35:23.999979 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:23.999496 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awjj7c" event={"ID":"ae9e9c00-2e1d-472a-b037-dd0ce0f229fd","Type":"ContainerDied","Data":"0754ced8a53a23d04fd31b4c59ba7479bc30daaf46da4b9eb73228d50bb01374"} Nov 03 20:35:23.999979 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:23.999628 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0754ced8a53a23d04fd31b4c59ba7479bc30daaf46da4b9eb73228d50bb01374" Nov 03 20:35:24.001586 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:24.001561 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb" event={"ID":"067cdd51-58e6-4823-a3ec-4e03ea366c58","Type":"ContainerDied","Data":"25ae0543b35722d4be028f693c05f6c9be6bc1b96c76dff2d891e23e415bfd60"} Nov 03 20:35:24.001688 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:24.001598 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25ae0543b35722d4be028f693c05f6c9be6bc1b96c76dff2d891e23e415bfd60" Nov 03 20:35:24.001688 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:24.001613 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp4mxb" Nov 03 20:35:36.306265 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.306173 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz"] Nov 03 20:35:36.306723 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.306543 4493 cpu_manager.go:414] "containerMap: removing container" podUID="067cdd51-58e6-4823-a3ec-4e03ea366c58" containerName="pull" Nov 03 20:35:36.306723 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.306556 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="067cdd51-58e6-4823-a3ec-4e03ea366c58" containerName="pull" Nov 03 20:35:36.306723 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.306572 4493 cpu_manager.go:414] "containerMap: removing container" podUID="4081add1-cd6a-48a0-aaa0-d76790121ecb" containerName="extract" Nov 03 20:35:36.306723 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.306576 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4081add1-cd6a-48a0-aaa0-d76790121ecb" containerName="extract" Nov 03 20:35:36.306723 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.306594 4493 cpu_manager.go:414] "containerMap: removing container" podUID="067cdd51-58e6-4823-a3ec-4e03ea366c58" containerName="extract" Nov 03 20:35:36.306723 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.306601 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="067cdd51-58e6-4823-a3ec-4e03ea366c58" containerName="extract" Nov 03 20:35:36.306723 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.306610 4493 cpu_manager.go:414] "containerMap: removing container" podUID="4081add1-cd6a-48a0-aaa0-d76790121ecb" containerName="util" Nov 03 20:35:36.306723 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.306616 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4081add1-cd6a-48a0-aaa0-d76790121ecb" containerName="util" Nov 03 20:35:36.306723 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.306623 4493 cpu_manager.go:414] "containerMap: removing container" podUID="4081add1-cd6a-48a0-aaa0-d76790121ecb" containerName="pull" Nov 03 20:35:36.306723 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.306627 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4081add1-cd6a-48a0-aaa0-d76790121ecb" containerName="pull" Nov 03 20:35:36.306723 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.306635 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ae9e9c00-2e1d-472a-b037-dd0ce0f229fd" containerName="pull" Nov 03 20:35:36.306723 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.306639 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae9e9c00-2e1d-472a-b037-dd0ce0f229fd" containerName="pull" Nov 03 20:35:36.306723 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.306646 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ae9e9c00-2e1d-472a-b037-dd0ce0f229fd" containerName="extract" Nov 03 20:35:36.306723 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.306655 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae9e9c00-2e1d-472a-b037-dd0ce0f229fd" containerName="extract" Nov 03 20:35:36.306723 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.306663 4493 cpu_manager.go:414] "containerMap: removing container" podUID="067cdd51-58e6-4823-a3ec-4e03ea366c58" containerName="util" Nov 03 20:35:36.306723 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.306669 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="067cdd51-58e6-4823-a3ec-4e03ea366c58" containerName="util" Nov 03 20:35:36.306723 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.306683 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ae9e9c00-2e1d-472a-b037-dd0ce0f229fd" containerName="util" Nov 03 20:35:36.306723 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.306687 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae9e9c00-2e1d-472a-b037-dd0ce0f229fd" containerName="util" Nov 03 20:35:36.307477 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.306741 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae9e9c00-2e1d-472a-b037-dd0ce0f229fd" containerName="extract" Nov 03 20:35:36.307477 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.306759 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="4081add1-cd6a-48a0-aaa0-d76790121ecb" containerName="extract" Nov 03 20:35:36.307477 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.306772 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="067cdd51-58e6-4823-a3ec-4e03ea366c58" containerName="extract" Nov 03 20:35:36.309196 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.309171 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" Nov 03 20:35:36.362815 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.362785 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5815ca47-705b-4abb-800d-224ae9cff4ff-manifests\") pod \"opendatahub-operator-controller-manager-768646756f-2zxpz\" (UID: \"5815ca47-705b-4abb-800d-224ae9cff4ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" Nov 03 20:35:36.362940 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.362874 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5815ca47-705b-4abb-800d-224ae9cff4ff-metrics-cert\") pod \"opendatahub-operator-controller-manager-768646756f-2zxpz\" (UID: \"5815ca47-705b-4abb-800d-224ae9cff4ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" Nov 03 20:35:36.362940 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.362922 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5815ca47-705b-4abb-800d-224ae9cff4ff-webhook-cert\") pod \"opendatahub-operator-controller-manager-768646756f-2zxpz\" (UID: \"5815ca47-705b-4abb-800d-224ae9cff4ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" Nov 03 20:35:36.363056 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.362948 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5815ca47-705b-4abb-800d-224ae9cff4ff-tmp\") pod \"opendatahub-operator-controller-manager-768646756f-2zxpz\" (UID: \"5815ca47-705b-4abb-800d-224ae9cff4ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" Nov 03 20:35:36.363056 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.362976 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-446cv\" (UniqueName: \"kubernetes.io/projected/5815ca47-705b-4abb-800d-224ae9cff4ff-kube-api-access-446cv\") pod \"opendatahub-operator-controller-manager-768646756f-2zxpz\" (UID: \"5815ca47-705b-4abb-800d-224ae9cff4ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" Nov 03 20:35:36.363056 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.362991 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 20:35:36.363056 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.362999 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5815ca47-705b-4abb-800d-224ae9cff4ff-apiservice-cert\") pod \"opendatahub-operator-controller-manager-768646756f-2zxpz\" (UID: \"5815ca47-705b-4abb-800d-224ae9cff4ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" Nov 03 20:35:36.363056 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.363042 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 20:35:36.363450 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.363434 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 20:35:36.364574 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.364556 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 20:35:36.364698 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.364681 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 20:35:36.410224 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.410195 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-s65qj\"" Nov 03 20:35:36.438911 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.438885 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz"] Nov 03 20:35:36.463883 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.463855 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-446cv\" (UniqueName: \"kubernetes.io/projected/5815ca47-705b-4abb-800d-224ae9cff4ff-kube-api-access-446cv\") pod \"opendatahub-operator-controller-manager-768646756f-2zxpz\" (UID: \"5815ca47-705b-4abb-800d-224ae9cff4ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" Nov 03 20:35:36.463986 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.463888 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5815ca47-705b-4abb-800d-224ae9cff4ff-apiservice-cert\") pod \"opendatahub-operator-controller-manager-768646756f-2zxpz\" (UID: \"5815ca47-705b-4abb-800d-224ae9cff4ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" Nov 03 20:35:36.463986 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.463920 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5815ca47-705b-4abb-800d-224ae9cff4ff-manifests\") pod \"opendatahub-operator-controller-manager-768646756f-2zxpz\" (UID: \"5815ca47-705b-4abb-800d-224ae9cff4ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" Nov 03 20:35:36.463986 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.463961 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5815ca47-705b-4abb-800d-224ae9cff4ff-metrics-cert\") pod \"opendatahub-operator-controller-manager-768646756f-2zxpz\" (UID: \"5815ca47-705b-4abb-800d-224ae9cff4ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" Nov 03 20:35:36.463986 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.463976 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5815ca47-705b-4abb-800d-224ae9cff4ff-webhook-cert\") pod \"opendatahub-operator-controller-manager-768646756f-2zxpz\" (UID: \"5815ca47-705b-4abb-800d-224ae9cff4ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" Nov 03 20:35:36.464191 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.463992 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5815ca47-705b-4abb-800d-224ae9cff4ff-tmp\") pod \"opendatahub-operator-controller-manager-768646756f-2zxpz\" (UID: \"5815ca47-705b-4abb-800d-224ae9cff4ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" Nov 03 20:35:36.464435 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.464417 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5815ca47-705b-4abb-800d-224ae9cff4ff-manifests\") pod \"opendatahub-operator-controller-manager-768646756f-2zxpz\" (UID: \"5815ca47-705b-4abb-800d-224ae9cff4ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" Nov 03 20:35:36.466409 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.466389 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5815ca47-705b-4abb-800d-224ae9cff4ff-tmp\") pod \"opendatahub-operator-controller-manager-768646756f-2zxpz\" (UID: \"5815ca47-705b-4abb-800d-224ae9cff4ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" Nov 03 20:35:36.466677 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.466654 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5815ca47-705b-4abb-800d-224ae9cff4ff-apiservice-cert\") pod \"opendatahub-operator-controller-manager-768646756f-2zxpz\" (UID: \"5815ca47-705b-4abb-800d-224ae9cff4ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" Nov 03 20:35:36.466901 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.466886 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5815ca47-705b-4abb-800d-224ae9cff4ff-webhook-cert\") pod \"opendatahub-operator-controller-manager-768646756f-2zxpz\" (UID: \"5815ca47-705b-4abb-800d-224ae9cff4ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" Nov 03 20:35:36.466952 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.466934 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5815ca47-705b-4abb-800d-224ae9cff4ff-metrics-cert\") pod \"opendatahub-operator-controller-manager-768646756f-2zxpz\" (UID: \"5815ca47-705b-4abb-800d-224ae9cff4ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" Nov 03 20:35:36.518051 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.518025 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-446cv\" (UniqueName: \"kubernetes.io/projected/5815ca47-705b-4abb-800d-224ae9cff4ff-kube-api-access-446cv\") pod \"opendatahub-operator-controller-manager-768646756f-2zxpz\" (UID: \"5815ca47-705b-4abb-800d-224ae9cff4ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" Nov 03 20:35:36.624354 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:36.624265 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" Nov 03 20:35:37.110307 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:37.110276 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz"] Nov 03 20:35:37.117640 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:35:37.117613 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5815ca47_705b_4abb_800d_224ae9cff4ff.slice/crio-d39c3ff554893b8b72343ff4fb1fe783a12b2c447d5ff5c27a142cdb08a5abe6 WatchSource:0}: Error finding container d39c3ff554893b8b72343ff4fb1fe783a12b2c447d5ff5c27a142cdb08a5abe6: Status 404 returned error can't find the container with id d39c3ff554893b8b72343ff4fb1fe783a12b2c447d5ff5c27a142cdb08a5abe6 Nov 03 20:35:38.051928 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:38.051891 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" event={"ID":"5815ca47-705b-4abb-800d-224ae9cff4ff","Type":"ContainerStarted","Data":"d39c3ff554893b8b72343ff4fb1fe783a12b2c447d5ff5c27a142cdb08a5abe6"} Nov 03 20:35:41.066159 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:41.066129 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" event={"ID":"5815ca47-705b-4abb-800d-224ae9cff4ff","Type":"ContainerStarted","Data":"a217c9d262e8b55629ec8f3932153238342dd894df63540ad74e81ac63ec80bf"} Nov 03 20:35:41.563769 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:41.563735 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-94q8x"] Nov 03 20:35:41.571885 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:41.571859 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-94q8x" Nov 03 20:35:41.609299 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:41.609268 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9p9c\" (UniqueName: \"kubernetes.io/projected/b4d26a60-6683-4bb3-b788-d0ed9d0e7732-kube-api-access-p9p9c\") pod \"jobset-operator-674cd59dc5-94q8x\" (UID: \"b4d26a60-6683-4bb3-b788-d0ed9d0e7732\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-94q8x" Nov 03 20:35:41.609476 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:41.609322 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4d26a60-6683-4bb3-b788-d0ed9d0e7732-tmp\") pod \"jobset-operator-674cd59dc5-94q8x\" (UID: \"b4d26a60-6683-4bb3-b788-d0ed9d0e7732\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-94q8x" Nov 03 20:35:41.621082 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:41.621034 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-94q8x"] Nov 03 20:35:41.621238 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:41.621132 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 20:35:41.621238 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:41.621225 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:35:41.663156 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:41.663138 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-dswcn\"" Nov 03 20:35:41.710498 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:41.710462 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4d26a60-6683-4bb3-b788-d0ed9d0e7732-tmp\") pod \"jobset-operator-674cd59dc5-94q8x\" (UID: \"b4d26a60-6683-4bb3-b788-d0ed9d0e7732\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-94q8x" Nov 03 20:35:41.710654 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:41.710529 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9p9c\" (UniqueName: \"kubernetes.io/projected/b4d26a60-6683-4bb3-b788-d0ed9d0e7732-kube-api-access-p9p9c\") pod \"jobset-operator-674cd59dc5-94q8x\" (UID: \"b4d26a60-6683-4bb3-b788-d0ed9d0e7732\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-94q8x" Nov 03 20:35:41.710910 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:41.710888 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4d26a60-6683-4bb3-b788-d0ed9d0e7732-tmp\") pod \"jobset-operator-674cd59dc5-94q8x\" (UID: \"b4d26a60-6683-4bb3-b788-d0ed9d0e7732\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-94q8x" Nov 03 20:35:41.766289 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:41.766251 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9p9c\" (UniqueName: \"kubernetes.io/projected/b4d26a60-6683-4bb3-b788-d0ed9d0e7732-kube-api-access-p9p9c\") pod \"jobset-operator-674cd59dc5-94q8x\" (UID: \"b4d26a60-6683-4bb3-b788-d0ed9d0e7732\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-94q8x" Nov 03 20:35:41.903755 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:41.903678 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-94q8x" Nov 03 20:35:42.072131 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:42.072093 4493 generic.go:356] "Generic (PLEG): container finished" podID="5815ca47-705b-4abb-800d-224ae9cff4ff" containerID="a217c9d262e8b55629ec8f3932153238342dd894df63540ad74e81ac63ec80bf" exitCode=0 Nov 03 20:35:42.072620 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:42.072136 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" event={"ID":"5815ca47-705b-4abb-800d-224ae9cff4ff","Type":"ContainerDied","Data":"a217c9d262e8b55629ec8f3932153238342dd894df63540ad74e81ac63ec80bf"} Nov 03 20:35:42.369566 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:42.369531 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-94q8x"] Nov 03 20:35:42.375998 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:35:42.375967 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4d26a60_6683_4bb3_b788_d0ed9d0e7732.slice/crio-441cda399d832f7ae19f2576626c0e500956690a51c19ad8d2ce3890d14ab2eb WatchSource:0}: Error finding container 441cda399d832f7ae19f2576626c0e500956690a51c19ad8d2ce3890d14ab2eb: Status 404 returned error can't find the container with id 441cda399d832f7ae19f2576626c0e500956690a51c19ad8d2ce3890d14ab2eb Nov 03 20:35:43.078528 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:43.078506 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-94q8x" event={"ID":"b4d26a60-6683-4bb3-b788-d0ed9d0e7732","Type":"ContainerStarted","Data":"441cda399d832f7ae19f2576626c0e500956690a51c19ad8d2ce3890d14ab2eb"} Nov 03 20:35:44.085084 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:44.085047 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" event={"ID":"5815ca47-705b-4abb-800d-224ae9cff4ff","Type":"ContainerStarted","Data":"a94026ef6bd1d89ea53fad9c3133d04a6091cb0bda37c80407cf67457fd419b4"} Nov 03 20:35:44.085596 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:44.085273 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" Nov 03 20:35:44.352869 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:44.352776 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" podStartSLOduration=3.4074386199999998 podStartE2EDuration="8.352760569s" totalImagesPullingTime="4.515450726s" totalInitContainerRuntime="429.871223ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:44.305409179 +0000 UTC m=+415.863399680" watchObservedRunningTime="2026-11-03 20:35:44.352760569 +0000 UTC m=+415.910751069" Nov 03 20:35:47.100606 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:47.100571 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-94q8x" event={"ID":"b4d26a60-6683-4bb3-b788-d0ed9d0e7732","Type":"ContainerStarted","Data":"25334c9d9dc09d95f092db35f54537e8a49650a5f11e24c1826a4fb628eae423"} Nov 03 20:35:47.212523 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:47.212422 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-94q8x" podStartSLOduration=1.643269175 podStartE2EDuration="6.212406276s" totalImagesPullingTime="4.569137101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:47.211523151 +0000 UTC m=+418.769513669" watchObservedRunningTime="2026-11-03 20:35:47.212406276 +0000 UTC m=+418.770396774" Nov 03 20:35:51.742835 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:35:51.742794 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:35:55.091637 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:55.091602 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-2zxpz" Nov 03 20:35:59.545636 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:59.545599 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-vnnqh"] Nov 03 20:35:59.547967 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:59.547950 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnnqh" Nov 03 20:35:59.605196 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:59.605173 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 20:35:59.647606 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:59.647585 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 20:35:59.647930 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:59.647912 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-gcjn6\"" Nov 03 20:35:59.648946 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:59.648931 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-vnnqh"] Nov 03 20:35:59.662417 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:59.662393 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d9b8\" (UniqueName: \"kubernetes.io/projected/524985f8-77ae-4383-bf88-355bde3b36ec-kube-api-access-6d9b8\") pod \"cert-manager-webhook-5dbbbf75cc-vnnqh\" (UID: \"524985f8-77ae-4383-bf88-355bde3b36ec\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnnqh" Nov 03 20:35:59.662533 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:59.662440 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/524985f8-77ae-4383-bf88-355bde3b36ec-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-vnnqh\" (UID: \"524985f8-77ae-4383-bf88-355bde3b36ec\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnnqh" Nov 03 20:35:59.763647 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:59.763610 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d9b8\" (UniqueName: \"kubernetes.io/projected/524985f8-77ae-4383-bf88-355bde3b36ec-kube-api-access-6d9b8\") pod \"cert-manager-webhook-5dbbbf75cc-vnnqh\" (UID: \"524985f8-77ae-4383-bf88-355bde3b36ec\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnnqh" Nov 03 20:35:59.763825 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:59.763676 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/524985f8-77ae-4383-bf88-355bde3b36ec-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-vnnqh\" (UID: \"524985f8-77ae-4383-bf88-355bde3b36ec\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnnqh" Nov 03 20:35:59.818138 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:59.818067 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/524985f8-77ae-4383-bf88-355bde3b36ec-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-vnnqh\" (UID: \"524985f8-77ae-4383-bf88-355bde3b36ec\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnnqh" Nov 03 20:35:59.818275 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:59.818225 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d9b8\" (UniqueName: \"kubernetes.io/projected/524985f8-77ae-4383-bf88-355bde3b36ec-kube-api-access-6d9b8\") pod \"cert-manager-webhook-5dbbbf75cc-vnnqh\" (UID: \"524985f8-77ae-4383-bf88-355bde3b36ec\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnnqh" Nov 03 20:35:59.858023 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:35:59.858000 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnnqh" Nov 03 20:36:00.226531 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:00.226455 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-vnnqh"] Nov 03 20:36:00.232772 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:36:00.232743 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod524985f8_77ae_4383_bf88_355bde3b36ec.slice/crio-5b83f22cc73019c019354bf1a5cab10f71c213540bb25a521376418c4dcd597e WatchSource:0}: Error finding container 5b83f22cc73019c019354bf1a5cab10f71c213540bb25a521376418c4dcd597e: Status 404 returned error can't find the container with id 5b83f22cc73019c019354bf1a5cab10f71c213540bb25a521376418c4dcd597e Nov 03 20:36:01.158723 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:01.158689 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnnqh" event={"ID":"524985f8-77ae-4383-bf88-355bde3b36ec","Type":"ContainerStarted","Data":"5b83f22cc73019c019354bf1a5cab10f71c213540bb25a521376418c4dcd597e"} Nov 03 20:36:05.179440 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:05.179398 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnnqh" event={"ID":"524985f8-77ae-4383-bf88-355bde3b36ec","Type":"ContainerStarted","Data":"ba2baa909977ad4594909567312dd3c438699001b0669ce23b0f5fde7adb2ac5"} Nov 03 20:36:05.179945 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:05.179511 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnnqh" Nov 03 20:36:05.292777 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:05.292717 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnnqh" podStartSLOduration=1.899896346 podStartE2EDuration="6.292703853s" totalImagesPullingTime="4.392807507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:05.291364316 +0000 UTC m=+436.849354837" watchObservedRunningTime="2026-11-03 20:36:05.292703853 +0000 UTC m=+436.850694352" Nov 03 20:36:05.751484 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:05.751456 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-dg9f6"] Nov 03 20:36:05.753770 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:05.753754 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dg9f6" Nov 03 20:36:05.804686 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:05.804657 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-7cjqq\"" Nov 03 20:36:05.809982 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:05.809959 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-dg9f6"] Nov 03 20:36:05.919409 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:05.919379 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxgtj\" (UniqueName: \"kubernetes.io/projected/4791bcc6-5240-4fb7-8c34-d1c7bd1cd8fa-kube-api-access-rxgtj\") pod \"cert-manager-cainjector-5c57bbd4fc-dg9f6\" (UID: \"4791bcc6-5240-4fb7-8c34-d1c7bd1cd8fa\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dg9f6" Nov 03 20:36:05.919579 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:05.919452 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4791bcc6-5240-4fb7-8c34-d1c7bd1cd8fa-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-dg9f6\" (UID: \"4791bcc6-5240-4fb7-8c34-d1c7bd1cd8fa\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dg9f6" Nov 03 20:36:06.020588 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:06.020510 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxgtj\" (UniqueName: \"kubernetes.io/projected/4791bcc6-5240-4fb7-8c34-d1c7bd1cd8fa-kube-api-access-rxgtj\") pod \"cert-manager-cainjector-5c57bbd4fc-dg9f6\" (UID: \"4791bcc6-5240-4fb7-8c34-d1c7bd1cd8fa\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dg9f6" Nov 03 20:36:06.020588 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:06.020579 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4791bcc6-5240-4fb7-8c34-d1c7bd1cd8fa-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-dg9f6\" (UID: \"4791bcc6-5240-4fb7-8c34-d1c7bd1cd8fa\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dg9f6" Nov 03 20:36:06.079211 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:06.079178 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4791bcc6-5240-4fb7-8c34-d1c7bd1cd8fa-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-dg9f6\" (UID: \"4791bcc6-5240-4fb7-8c34-d1c7bd1cd8fa\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dg9f6" Nov 03 20:36:06.079427 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:06.079407 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxgtj\" (UniqueName: \"kubernetes.io/projected/4791bcc6-5240-4fb7-8c34-d1c7bd1cd8fa-kube-api-access-rxgtj\") pod \"cert-manager-cainjector-5c57bbd4fc-dg9f6\" (UID: \"4791bcc6-5240-4fb7-8c34-d1c7bd1cd8fa\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dg9f6" Nov 03 20:36:06.364112 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:06.364078 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dg9f6" Nov 03 20:36:06.747509 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:06.747438 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-dg9f6"] Nov 03 20:36:06.753426 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:36:06.753386 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4791bcc6_5240_4fb7_8c34_d1c7bd1cd8fa.slice/crio-285e8212a329128835ebcccd6909c7482f6f0ce38e657a81f17ebf60c9abe949 WatchSource:0}: Error finding container 285e8212a329128835ebcccd6909c7482f6f0ce38e657a81f17ebf60c9abe949: Status 404 returned error can't find the container with id 285e8212a329128835ebcccd6909c7482f6f0ce38e657a81f17ebf60c9abe949 Nov 03 20:36:07.189087 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:07.189056 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dg9f6" event={"ID":"4791bcc6-5240-4fb7-8c34-d1c7bd1cd8fa","Type":"ContainerStarted","Data":"5eaaa13f8d3e0048d196858eb1ff0e19765311ab1d1e7bed63074693369a0051"} Nov 03 20:36:07.189257 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:07.189097 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dg9f6" event={"ID":"4791bcc6-5240-4fb7-8c34-d1c7bd1cd8fa","Type":"ContainerStarted","Data":"285e8212a329128835ebcccd6909c7482f6f0ce38e657a81f17ebf60c9abe949"} Nov 03 20:36:07.296387 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:07.296323 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dg9f6" podStartSLOduration=2.29630966 podStartE2EDuration="2.29630966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:07.296015368 +0000 UTC m=+438.854005885" watchObservedRunningTime="2026-11-03 20:36:07.29630966 +0000 UTC m=+438.854300159" Nov 03 20:36:07.917692 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:07.917660 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-5k9hh"] Nov 03 20:36:07.920115 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:07.920094 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5k9hh" Nov 03 20:36:07.969506 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:07.969486 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 03 20:36:07.969635 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:07.969512 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 20:36:08.016851 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:08.016822 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 20:36:08.018271 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:08.018248 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-5k9hh"] Nov 03 20:36:08.035582 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:08.035555 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd5hs\" (UniqueName: \"kubernetes.io/projected/eb2d5dbe-0f4d-4b1e-88b9-913c6df7d20a-kube-api-access-xd5hs\") pod \"kuadrant-console-plugin-66c4fb575d-5k9hh\" (UID: \"eb2d5dbe-0f4d-4b1e-88b9-913c6df7d20a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5k9hh" Nov 03 20:36:08.035731 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:08.035609 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb2d5dbe-0f4d-4b1e-88b9-913c6df7d20a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-5k9hh\" (UID: \"eb2d5dbe-0f4d-4b1e-88b9-913c6df7d20a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5k9hh" Nov 03 20:36:08.035778 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:08.035729 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eb2d5dbe-0f4d-4b1e-88b9-913c6df7d20a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-5k9hh\" (UID: \"eb2d5dbe-0f4d-4b1e-88b9-913c6df7d20a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5k9hh" Nov 03 20:36:08.064008 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:08.063984 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 03 20:36:08.064008 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:08.064002 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-n792z\"" Nov 03 20:36:08.137303 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:08.137275 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd5hs\" (UniqueName: \"kubernetes.io/projected/eb2d5dbe-0f4d-4b1e-88b9-913c6df7d20a-kube-api-access-xd5hs\") pod \"kuadrant-console-plugin-66c4fb575d-5k9hh\" (UID: \"eb2d5dbe-0f4d-4b1e-88b9-913c6df7d20a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5k9hh" Nov 03 20:36:08.137463 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:08.137328 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb2d5dbe-0f4d-4b1e-88b9-913c6df7d20a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-5k9hh\" (UID: \"eb2d5dbe-0f4d-4b1e-88b9-913c6df7d20a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5k9hh" Nov 03 20:36:08.137463 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:08.137389 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eb2d5dbe-0f4d-4b1e-88b9-913c6df7d20a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-5k9hh\" (UID: \"eb2d5dbe-0f4d-4b1e-88b9-913c6df7d20a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5k9hh" Nov 03 20:36:08.138034 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:08.138009 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eb2d5dbe-0f4d-4b1e-88b9-913c6df7d20a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-5k9hh\" (UID: \"eb2d5dbe-0f4d-4b1e-88b9-913c6df7d20a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5k9hh" Nov 03 20:36:08.140087 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:08.140061 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb2d5dbe-0f4d-4b1e-88b9-913c6df7d20a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-5k9hh\" (UID: \"eb2d5dbe-0f4d-4b1e-88b9-913c6df7d20a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5k9hh" Nov 03 20:36:08.191786 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:08.191715 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd5hs\" (UniqueName: \"kubernetes.io/projected/eb2d5dbe-0f4d-4b1e-88b9-913c6df7d20a-kube-api-access-xd5hs\") pod \"kuadrant-console-plugin-66c4fb575d-5k9hh\" (UID: \"eb2d5dbe-0f4d-4b1e-88b9-913c6df7d20a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5k9hh" Nov 03 20:36:08.230854 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:08.230828 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5k9hh" Nov 03 20:36:08.596541 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:08.596507 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-5k9hh"] Nov 03 20:36:08.602767 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:36:08.602744 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb2d5dbe_0f4d_4b1e_88b9_913c6df7d20a.slice/crio-6ae5bf90679795f6015230c95859704393cba3f3ade449d7099198167d93c18a WatchSource:0}: Error finding container 6ae5bf90679795f6015230c95859704393cba3f3ade449d7099198167d93c18a: Status 404 returned error can't find the container with id 6ae5bf90679795f6015230c95859704393cba3f3ade449d7099198167d93c18a Nov 03 20:36:09.198214 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:09.198183 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5k9hh" event={"ID":"eb2d5dbe-0f4d-4b1e-88b9-913c6df7d20a","Type":"ContainerStarted","Data":"6ae5bf90679795f6015230c95859704393cba3f3ade449d7099198167d93c18a"} Nov 03 20:36:09.237644 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:09.237616 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-7b5lx"] Nov 03 20:36:09.240699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:09.240685 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-7b5lx" Nov 03 20:36:09.297018 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:09.296994 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-7b5lx"] Nov 03 20:36:09.339518 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:09.339493 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-tzs9k\"" Nov 03 20:36:09.346845 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:09.346815 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7830cc7a-a8af-4fef-bb56-e6bc23ac2a47-bound-sa-token\") pod \"cert-manager-597b95cb84-7b5lx\" (UID: \"7830cc7a-a8af-4fef-bb56-e6bc23ac2a47\") " pod="cert-manager/cert-manager-597b95cb84-7b5lx" Nov 03 20:36:09.347036 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:09.346864 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtgwn\" (UniqueName: \"kubernetes.io/projected/7830cc7a-a8af-4fef-bb56-e6bc23ac2a47-kube-api-access-dtgwn\") pod \"cert-manager-597b95cb84-7b5lx\" (UID: \"7830cc7a-a8af-4fef-bb56-e6bc23ac2a47\") " pod="cert-manager/cert-manager-597b95cb84-7b5lx" Nov 03 20:36:09.447593 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:09.447561 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7830cc7a-a8af-4fef-bb56-e6bc23ac2a47-bound-sa-token\") pod \"cert-manager-597b95cb84-7b5lx\" (UID: \"7830cc7a-a8af-4fef-bb56-e6bc23ac2a47\") " pod="cert-manager/cert-manager-597b95cb84-7b5lx" Nov 03 20:36:09.447773 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:09.447623 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtgwn\" (UniqueName: \"kubernetes.io/projected/7830cc7a-a8af-4fef-bb56-e6bc23ac2a47-kube-api-access-dtgwn\") pod \"cert-manager-597b95cb84-7b5lx\" (UID: \"7830cc7a-a8af-4fef-bb56-e6bc23ac2a47\") " pod="cert-manager/cert-manager-597b95cb84-7b5lx" Nov 03 20:36:09.501361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:09.501285 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7830cc7a-a8af-4fef-bb56-e6bc23ac2a47-bound-sa-token\") pod \"cert-manager-597b95cb84-7b5lx\" (UID: \"7830cc7a-a8af-4fef-bb56-e6bc23ac2a47\") " pod="cert-manager/cert-manager-597b95cb84-7b5lx" Nov 03 20:36:09.501595 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:09.501573 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtgwn\" (UniqueName: \"kubernetes.io/projected/7830cc7a-a8af-4fef-bb56-e6bc23ac2a47-kube-api-access-dtgwn\") pod \"cert-manager-597b95cb84-7b5lx\" (UID: \"7830cc7a-a8af-4fef-bb56-e6bc23ac2a47\") " pod="cert-manager/cert-manager-597b95cb84-7b5lx" Nov 03 20:36:09.551073 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:09.551048 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-7b5lx" Nov 03 20:36:09.914726 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:09.914691 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-7b5lx"] Nov 03 20:36:09.921154 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:36:09.921124 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7830cc7a_a8af_4fef_bb56_e6bc23ac2a47.slice/crio-7fb47696cce80fa7e9c553e844f2fe9cfb9d7e2e24367f1246e73f029ee1a85c WatchSource:0}: Error finding container 7fb47696cce80fa7e9c553e844f2fe9cfb9d7e2e24367f1246e73f029ee1a85c: Status 404 returned error can't find the container with id 7fb47696cce80fa7e9c553e844f2fe9cfb9d7e2e24367f1246e73f029ee1a85c Nov 03 20:36:10.203865 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:10.203790 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-7b5lx" event={"ID":"7830cc7a-a8af-4fef-bb56-e6bc23ac2a47","Type":"ContainerStarted","Data":"0b4d6b23642146c4ecda43a07bc7a18b9d26eccb7f8d0903b68e001db00ece74"} Nov 03 20:36:10.203865 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:10.203826 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-7b5lx" event={"ID":"7830cc7a-a8af-4fef-bb56-e6bc23ac2a47","Type":"ContainerStarted","Data":"7fb47696cce80fa7e9c553e844f2fe9cfb9d7e2e24367f1246e73f029ee1a85c"} Nov 03 20:36:10.311837 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:10.311782 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-7b5lx" podStartSLOduration=1.311765745 podStartE2EDuration="1.311765745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:10.310712872 +0000 UTC m=+441.868703390" watchObservedRunningTime="2026-11-03 20:36:10.311765745 +0000 UTC m=+441.869756243" Nov 03 20:36:11.185891 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:11.185860 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnnqh" Nov 03 20:36:16.192274 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:16.192239 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-bb87d7467-shtb7"] Nov 03 20:36:16.199963 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:16.199941 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-bb87d7467-shtb7" Nov 03 20:36:16.236517 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:16.236485 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5k9hh" event={"ID":"eb2d5dbe-0f4d-4b1e-88b9-913c6df7d20a","Type":"ContainerStarted","Data":"289a52a09e64bf888f3dded5e96e2d89e7d399198a8b8081d29cd978e4a07c5c"} Nov 03 20:36:16.249220 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:16.249199 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ccl98\"" Nov 03 20:36:16.296634 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:16.296603 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 20:36:16.296753 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:16.296633 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 20:36:16.297975 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:16.297955 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-bb87d7467-shtb7"] Nov 03 20:36:16.308978 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:16.308961 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59c188bf-2663-4d7d-8b1a-cd3320f45b28-tmp\") pod \"kube-auth-proxy-bb87d7467-shtb7\" (UID: \"59c188bf-2663-4d7d-8b1a-cd3320f45b28\") " pod="openshift-ingress/kube-auth-proxy-bb87d7467-shtb7" Nov 03 20:36:16.309040 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:16.308996 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59c188bf-2663-4d7d-8b1a-cd3320f45b28-tls-certs\") pod \"kube-auth-proxy-bb87d7467-shtb7\" (UID: \"59c188bf-2663-4d7d-8b1a-cd3320f45b28\") " pod="openshift-ingress/kube-auth-proxy-bb87d7467-shtb7" Nov 03 20:36:16.309040 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:16.309020 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j5zd\" (UniqueName: \"kubernetes.io/projected/59c188bf-2663-4d7d-8b1a-cd3320f45b28-kube-api-access-5j5zd\") pod \"kube-auth-proxy-bb87d7467-shtb7\" (UID: \"59c188bf-2663-4d7d-8b1a-cd3320f45b28\") " pod="openshift-ingress/kube-auth-proxy-bb87d7467-shtb7" Nov 03 20:36:16.343490 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:16.343467 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 20:36:16.343490 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:16.343473 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 20:36:16.409724 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:16.409694 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59c188bf-2663-4d7d-8b1a-cd3320f45b28-tls-certs\") pod \"kube-auth-proxy-bb87d7467-shtb7\" (UID: \"59c188bf-2663-4d7d-8b1a-cd3320f45b28\") " pod="openshift-ingress/kube-auth-proxy-bb87d7467-shtb7" Nov 03 20:36:16.409881 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:16.409744 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j5zd\" (UniqueName: \"kubernetes.io/projected/59c188bf-2663-4d7d-8b1a-cd3320f45b28-kube-api-access-5j5zd\") pod \"kube-auth-proxy-bb87d7467-shtb7\" (UID: \"59c188bf-2663-4d7d-8b1a-cd3320f45b28\") " pod="openshift-ingress/kube-auth-proxy-bb87d7467-shtb7" Nov 03 20:36:16.409881 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:16.409842 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59c188bf-2663-4d7d-8b1a-cd3320f45b28-tmp\") pod \"kube-auth-proxy-bb87d7467-shtb7\" (UID: \"59c188bf-2663-4d7d-8b1a-cd3320f45b28\") " pod="openshift-ingress/kube-auth-proxy-bb87d7467-shtb7" Nov 03 20:36:16.412787 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:16.412763 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59c188bf-2663-4d7d-8b1a-cd3320f45b28-tmp\") pod \"kube-auth-proxy-bb87d7467-shtb7\" (UID: \"59c188bf-2663-4d7d-8b1a-cd3320f45b28\") " pod="openshift-ingress/kube-auth-proxy-bb87d7467-shtb7" Nov 03 20:36:16.413013 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:16.412997 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59c188bf-2663-4d7d-8b1a-cd3320f45b28-tls-certs\") pod \"kube-auth-proxy-bb87d7467-shtb7\" (UID: \"59c188bf-2663-4d7d-8b1a-cd3320f45b28\") " pod="openshift-ingress/kube-auth-proxy-bb87d7467-shtb7" Nov 03 20:36:16.423416 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:16.423377 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5k9hh" podStartSLOduration=2.821420814 podStartE2EDuration="9.423365076s" totalImagesPullingTime="6.601944262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:16.422140052 +0000 UTC m=+447.980130573" watchObservedRunningTime="2026-11-03 20:36:16.423365076 +0000 UTC m=+447.981355574" Nov 03 20:36:16.463668 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:16.463599 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j5zd\" (UniqueName: \"kubernetes.io/projected/59c188bf-2663-4d7d-8b1a-cd3320f45b28-kube-api-access-5j5zd\") pod \"kube-auth-proxy-bb87d7467-shtb7\" (UID: \"59c188bf-2663-4d7d-8b1a-cd3320f45b28\") " pod="openshift-ingress/kube-auth-proxy-bb87d7467-shtb7" Nov 03 20:36:16.511603 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:16.511576 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-bb87d7467-shtb7" Nov 03 20:36:16.878059 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:16.878029 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-bb87d7467-shtb7"] Nov 03 20:36:16.884746 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:36:16.884724 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c188bf_2663_4d7d_8b1a_cd3320f45b28.slice/crio-0192a96ccb74640022e6accd4e31e33e6ab5aa3c3010f97cd20f8bd4f2a7c876 WatchSource:0}: Error finding container 0192a96ccb74640022e6accd4e31e33e6ab5aa3c3010f97cd20f8bd4f2a7c876: Status 404 returned error can't find the container with id 0192a96ccb74640022e6accd4e31e33e6ab5aa3c3010f97cd20f8bd4f2a7c876 Nov 03 20:36:17.242457 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:17.242361 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-bb87d7467-shtb7" event={"ID":"59c188bf-2663-4d7d-8b1a-cd3320f45b28","Type":"ContainerStarted","Data":"0192a96ccb74640022e6accd4e31e33e6ab5aa3c3010f97cd20f8bd4f2a7c876"} Nov 03 20:36:21.747110 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:36:21.747075 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:36:22.266894 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:22.266863 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-bb87d7467-shtb7" event={"ID":"59c188bf-2663-4d7d-8b1a-cd3320f45b28","Type":"ContainerStarted","Data":"63d47d700a076e617c2ab9c7cf906e09052214e8d32c25ad12ff98d8629a2eb2"} Nov 03 20:36:22.375848 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:22.375794 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-bb87d7467-shtb7" podStartSLOduration=1.54351271 podStartE2EDuration="6.375780819s" totalImagesPullingTime="4.832268109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:22.374322088 +0000 UTC m=+453.932312600" watchObservedRunningTime="2026-11-03 20:36:22.375780819 +0000 UTC m=+453.933771320" Nov 03 20:36:27.752022 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:27.751990 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-49ddb"] Nov 03 20:36:27.755588 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:27.755568 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-49ddb" Nov 03 20:36:27.796174 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:27.796144 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kk8q\" (UniqueName: \"kubernetes.io/projected/07dea7e4-f5d0-485f-ad61-28b3ac0d9f85-kube-api-access-8kk8q\") pod \"kserve-module-controller-manager-7566cf5fc8-49ddb\" (UID: \"07dea7e4-f5d0-485f-ad61-28b3ac0d9f85\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-49ddb" Nov 03 20:36:27.796277 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:27.796183 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/07dea7e4-f5d0-485f-ad61-28b3ac0d9f85-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-49ddb\" (UID: \"07dea7e4-f5d0-485f-ad61-28b3ac0d9f85\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-49ddb" Nov 03 20:36:27.796317 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:27.796299 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07dea7e4-f5d0-485f-ad61-28b3ac0d9f85-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-49ddb\" (UID: \"07dea7e4-f5d0-485f-ad61-28b3ac0d9f85\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-49ddb" Nov 03 20:36:27.796380 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:27.796349 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07dea7e4-f5d0-485f-ad61-28b3ac0d9f85-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-49ddb\" (UID: \"07dea7e4-f5d0-485f-ad61-28b3ac0d9f85\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-49ddb" Nov 03 20:36:27.805267 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:27.805249 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 20:36:27.805400 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:27.805385 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 20:36:27.852297 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:27.852265 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-mgdhx\"" Nov 03 20:36:27.897001 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:27.896974 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/07dea7e4-f5d0-485f-ad61-28b3ac0d9f85-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-49ddb\" (UID: \"07dea7e4-f5d0-485f-ad61-28b3ac0d9f85\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-49ddb" Nov 03 20:36:27.897188 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:27.897016 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07dea7e4-f5d0-485f-ad61-28b3ac0d9f85-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-49ddb\" (UID: \"07dea7e4-f5d0-485f-ad61-28b3ac0d9f85\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-49ddb" Nov 03 20:36:27.897188 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:27.897039 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07dea7e4-f5d0-485f-ad61-28b3ac0d9f85-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-49ddb\" (UID: \"07dea7e4-f5d0-485f-ad61-28b3ac0d9f85\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-49ddb" Nov 03 20:36:27.897188 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:27.897155 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kk8q\" (UniqueName: \"kubernetes.io/projected/07dea7e4-f5d0-485f-ad61-28b3ac0d9f85-kube-api-access-8kk8q\") pod \"kserve-module-controller-manager-7566cf5fc8-49ddb\" (UID: \"07dea7e4-f5d0-485f-ad61-28b3ac0d9f85\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-49ddb" Nov 03 20:36:27.897396 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:27.897376 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07dea7e4-f5d0-485f-ad61-28b3ac0d9f85-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-49ddb\" (UID: \"07dea7e4-f5d0-485f-ad61-28b3ac0d9f85\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-49ddb" Nov 03 20:36:27.897484 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:27.897466 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/07dea7e4-f5d0-485f-ad61-28b3ac0d9f85-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-49ddb\" (UID: \"07dea7e4-f5d0-485f-ad61-28b3ac0d9f85\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-49ddb" Nov 03 20:36:27.947372 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:27.947323 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 20:36:27.948774 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:27.948757 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-49ddb"] Nov 03 20:36:27.953642 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:27.953616 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kk8q\" (UniqueName: \"kubernetes.io/projected/07dea7e4-f5d0-485f-ad61-28b3ac0d9f85-kube-api-access-8kk8q\") pod \"kserve-module-controller-manager-7566cf5fc8-49ddb\" (UID: \"07dea7e4-f5d0-485f-ad61-28b3ac0d9f85\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-49ddb" Nov 03 20:36:27.958342 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:27.958308 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07dea7e4-f5d0-485f-ad61-28b3ac0d9f85-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-49ddb\" (UID: \"07dea7e4-f5d0-485f-ad61-28b3ac0d9f85\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-49ddb" Nov 03 20:36:28.067874 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:28.067855 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-49ddb" Nov 03 20:36:28.485238 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:28.485164 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-49ddb"] Nov 03 20:36:28.491733 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:36:28.491705 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07dea7e4_f5d0_485f_ad61_28b3ac0d9f85.slice/crio-0f6234f2c032006e1de7ed5c249a666fdbb9fff47fd86e4e5e53405b0bd71784 WatchSource:0}: Error finding container 0f6234f2c032006e1de7ed5c249a666fdbb9fff47fd86e4e5e53405b0bd71784: Status 404 returned error can't find the container with id 0f6234f2c032006e1de7ed5c249a666fdbb9fff47fd86e4e5e53405b0bd71784 Nov 03 20:36:29.297926 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:29.297894 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-49ddb" event={"ID":"07dea7e4-f5d0-485f-ad61-28b3ac0d9f85","Type":"ContainerStarted","Data":"0f6234f2c032006e1de7ed5c249a666fdbb9fff47fd86e4e5e53405b0bd71784"} Nov 03 20:36:31.308481 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:31.308443 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-49ddb" event={"ID":"07dea7e4-f5d0-485f-ad61-28b3ac0d9f85","Type":"ContainerStarted","Data":"bcb90732e5e7c58cc859be77e2b288432f11b6a1242014b477f46fe14616a6d0"} Nov 03 20:36:31.308958 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:31.308517 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-49ddb" Nov 03 20:36:31.418296 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:31.418238 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-49ddb" podStartSLOduration=2.28345205 podStartE2EDuration="4.418225092s" totalImagesPullingTime="2.134773042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:31.417384585 +0000 UTC m=+462.975375096" watchObservedRunningTime="2026-11-03 20:36:31.418225092 +0000 UTC m=+462.976215591" Nov 03 20:36:44.537520 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:44.537487 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5f45f855b6-gjwbt"] Nov 03 20:36:44.586498 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:44.586462 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5f45f855b6-gjwbt" Nov 03 20:36:44.604748 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:44.604716 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5f45f855b6-gjwbt"] Nov 03 20:36:44.638307 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:44.638267 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b92e805a-8634-435a-b8a1-159b4cf916e4-cert\") pod \"jobset-controller-manager-5f45f855b6-gjwbt\" (UID: \"b92e805a-8634-435a-b8a1-159b4cf916e4\") " pod="openshift-jobset-operator/jobset-controller-manager-5f45f855b6-gjwbt" Nov 03 20:36:44.638466 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:44.638418 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b92e805a-8634-435a-b8a1-159b4cf916e4-manager-config\") pod \"jobset-controller-manager-5f45f855b6-gjwbt\" (UID: \"b92e805a-8634-435a-b8a1-159b4cf916e4\") " pod="openshift-jobset-operator/jobset-controller-manager-5f45f855b6-gjwbt" Nov 03 20:36:44.638466 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:44.638463 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b92e805a-8634-435a-b8a1-159b4cf916e4-metrics-certs\") pod \"jobset-controller-manager-5f45f855b6-gjwbt\" (UID: \"b92e805a-8634-435a-b8a1-159b4cf916e4\") " pod="openshift-jobset-operator/jobset-controller-manager-5f45f855b6-gjwbt" Nov 03 20:36:44.638543 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:44.638487 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22jvv\" (UniqueName: \"kubernetes.io/projected/b92e805a-8634-435a-b8a1-159b4cf916e4-kube-api-access-22jvv\") pod \"jobset-controller-manager-5f45f855b6-gjwbt\" (UID: \"b92e805a-8634-435a-b8a1-159b4cf916e4\") " pod="openshift-jobset-operator/jobset-controller-manager-5f45f855b6-gjwbt" Nov 03 20:36:44.683187 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:44.683157 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-v2pbb\"" Nov 03 20:36:44.683399 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:44.683384 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 03 20:36:44.683530 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:44.683396 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 03 20:36:44.683530 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:44.683460 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 03 20:36:44.738916 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:44.738887 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b92e805a-8634-435a-b8a1-159b4cf916e4-metrics-certs\") pod \"jobset-controller-manager-5f45f855b6-gjwbt\" (UID: \"b92e805a-8634-435a-b8a1-159b4cf916e4\") " pod="openshift-jobset-operator/jobset-controller-manager-5f45f855b6-gjwbt" Nov 03 20:36:44.739047 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:44.738920 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22jvv\" (UniqueName: \"kubernetes.io/projected/b92e805a-8634-435a-b8a1-159b4cf916e4-kube-api-access-22jvv\") pod \"jobset-controller-manager-5f45f855b6-gjwbt\" (UID: \"b92e805a-8634-435a-b8a1-159b4cf916e4\") " pod="openshift-jobset-operator/jobset-controller-manager-5f45f855b6-gjwbt" Nov 03 20:36:44.739047 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:44.738958 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b92e805a-8634-435a-b8a1-159b4cf916e4-cert\") pod \"jobset-controller-manager-5f45f855b6-gjwbt\" (UID: \"b92e805a-8634-435a-b8a1-159b4cf916e4\") " pod="openshift-jobset-operator/jobset-controller-manager-5f45f855b6-gjwbt" Nov 03 20:36:44.739047 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:44.738990 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b92e805a-8634-435a-b8a1-159b4cf916e4-manager-config\") pod \"jobset-controller-manager-5f45f855b6-gjwbt\" (UID: \"b92e805a-8634-435a-b8a1-159b4cf916e4\") " pod="openshift-jobset-operator/jobset-controller-manager-5f45f855b6-gjwbt" Nov 03 20:36:44.739709 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:44.739685 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b92e805a-8634-435a-b8a1-159b4cf916e4-manager-config\") pod \"jobset-controller-manager-5f45f855b6-gjwbt\" (UID: \"b92e805a-8634-435a-b8a1-159b4cf916e4\") " pod="openshift-jobset-operator/jobset-controller-manager-5f45f855b6-gjwbt" Nov 03 20:36:44.741460 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:44.741440 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b92e805a-8634-435a-b8a1-159b4cf916e4-metrics-certs\") pod \"jobset-controller-manager-5f45f855b6-gjwbt\" (UID: \"b92e805a-8634-435a-b8a1-159b4cf916e4\") " pod="openshift-jobset-operator/jobset-controller-manager-5f45f855b6-gjwbt" Nov 03 20:36:44.741547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:44.741482 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b92e805a-8634-435a-b8a1-159b4cf916e4-cert\") pod \"jobset-controller-manager-5f45f855b6-gjwbt\" (UID: \"b92e805a-8634-435a-b8a1-159b4cf916e4\") " pod="openshift-jobset-operator/jobset-controller-manager-5f45f855b6-gjwbt" Nov 03 20:36:44.793190 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:44.793133 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22jvv\" (UniqueName: \"kubernetes.io/projected/b92e805a-8634-435a-b8a1-159b4cf916e4-kube-api-access-22jvv\") pod \"jobset-controller-manager-5f45f855b6-gjwbt\" (UID: \"b92e805a-8634-435a-b8a1-159b4cf916e4\") " pod="openshift-jobset-operator/jobset-controller-manager-5f45f855b6-gjwbt" Nov 03 20:36:44.954972 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:44.954926 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5f45f855b6-gjwbt" Nov 03 20:36:45.374576 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:45.374490 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5f45f855b6-gjwbt"] Nov 03 20:36:45.380696 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:36:45.380661 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92e805a_8634_435a_b8a1_159b4cf916e4.slice/crio-e93312b46e53c5254ecda40d8606b690726c8f6617530b83b4ee128a599956da WatchSource:0}: Error finding container e93312b46e53c5254ecda40d8606b690726c8f6617530b83b4ee128a599956da: Status 404 returned error can't find the container with id e93312b46e53c5254ecda40d8606b690726c8f6617530b83b4ee128a599956da Nov 03 20:36:46.377137 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:46.377105 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5f45f855b6-gjwbt" event={"ID":"b92e805a-8634-435a-b8a1-159b4cf916e4","Type":"ContainerStarted","Data":"e93312b46e53c5254ecda40d8606b690726c8f6617530b83b4ee128a599956da"} Nov 03 20:36:51.743262 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:36:51.743213 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:36:52.314587 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:52.314558 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-49ddb" Nov 03 20:36:52.406356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:52.406252 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5f45f855b6-gjwbt" event={"ID":"b92e805a-8634-435a-b8a1-159b4cf916e4","Type":"ContainerStarted","Data":"c8130374a9e2e310870b7a3302129628fd75159442d4289e10f8c21a70e1a27b"} Nov 03 20:36:52.406356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:52.406310 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5f45f855b6-gjwbt" Nov 03 20:36:52.579266 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:36:52.579205 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5f45f855b6-gjwbt" podStartSLOduration=1.837474131 podStartE2EDuration="8.579191342s" totalImagesPullingTime="6.741717211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:52.577714839 +0000 UTC m=+484.135705339" watchObservedRunningTime="2026-11-03 20:36:52.579191342 +0000 UTC m=+484.137181840" Nov 03 20:37:03.416324 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:37:03.416294 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5f45f855b6-gjwbt" Nov 03 20:37:21.743781 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:37:21.743746 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:37:51.743620 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:37:51.743581 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:38:21.743441 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:38:21.743318 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:38:36.969538 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:36.969458 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn"] Nov 03 20:38:36.972784 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:36.972765 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn" Nov 03 20:38:37.022559 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:37.022531 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:38:37.069624 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:37.069598 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:38:37.069765 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:37.069608 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:38:37.069765 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:37.069609 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nqbhl\"" Nov 03 20:38:37.069765 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:37.069634 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 20:38:37.071098 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:37.071081 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn"] Nov 03 20:38:37.075993 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:37.075970 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feae879c-5eba-47e8-bb99-c6bcd5939ab0-maas-api-tls\") pod \"maas-api-5ff6ff6f8b-2blhn\" (UID: \"feae879c-5eba-47e8-bb99-c6bcd5939ab0\") " pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn" Nov 03 20:38:37.076099 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:37.076019 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feae879c-5eba-47e8-bb99-c6bcd5939ab0-metrics-certs\") pod \"maas-api-5ff6ff6f8b-2blhn\" (UID: \"feae879c-5eba-47e8-bb99-c6bcd5939ab0\") " pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn" Nov 03 20:38:37.076099 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:37.076043 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feae879c-5eba-47e8-bb99-c6bcd5939ab0-openshift-service-ca\") pod \"maas-api-5ff6ff6f8b-2blhn\" (UID: \"feae879c-5eba-47e8-bb99-c6bcd5939ab0\") " pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn" Nov 03 20:38:37.076099 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:37.076088 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjpv5\" (UniqueName: \"kubernetes.io/projected/feae879c-5eba-47e8-bb99-c6bcd5939ab0-kube-api-access-pjpv5\") pod \"maas-api-5ff6ff6f8b-2blhn\" (UID: \"feae879c-5eba-47e8-bb99-c6bcd5939ab0\") " pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn" Nov 03 20:38:37.176738 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:37.176704 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feae879c-5eba-47e8-bb99-c6bcd5939ab0-maas-api-tls\") pod \"maas-api-5ff6ff6f8b-2blhn\" (UID: \"feae879c-5eba-47e8-bb99-c6bcd5939ab0\") " pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn" Nov 03 20:38:37.176916 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:37.176767 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feae879c-5eba-47e8-bb99-c6bcd5939ab0-metrics-certs\") pod \"maas-api-5ff6ff6f8b-2blhn\" (UID: \"feae879c-5eba-47e8-bb99-c6bcd5939ab0\") " pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn" Nov 03 20:38:37.176991 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:37.176924 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feae879c-5eba-47e8-bb99-c6bcd5939ab0-openshift-service-ca\") pod \"maas-api-5ff6ff6f8b-2blhn\" (UID: \"feae879c-5eba-47e8-bb99-c6bcd5939ab0\") " pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn" Nov 03 20:38:37.176991 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:37.176949 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjpv5\" (UniqueName: \"kubernetes.io/projected/feae879c-5eba-47e8-bb99-c6bcd5939ab0-kube-api-access-pjpv5\") pod \"maas-api-5ff6ff6f8b-2blhn\" (UID: \"feae879c-5eba-47e8-bb99-c6bcd5939ab0\") " pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn" Nov 03 20:38:37.177555 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:37.177528 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feae879c-5eba-47e8-bb99-c6bcd5939ab0-openshift-service-ca\") pod \"maas-api-5ff6ff6f8b-2blhn\" (UID: \"feae879c-5eba-47e8-bb99-c6bcd5939ab0\") " pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn" Nov 03 20:38:37.179467 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:37.179450 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feae879c-5eba-47e8-bb99-c6bcd5939ab0-metrics-certs\") pod \"maas-api-5ff6ff6f8b-2blhn\" (UID: \"feae879c-5eba-47e8-bb99-c6bcd5939ab0\") " pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn" Nov 03 20:38:37.179545 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:37.179467 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feae879c-5eba-47e8-bb99-c6bcd5939ab0-maas-api-tls\") pod \"maas-api-5ff6ff6f8b-2blhn\" (UID: \"feae879c-5eba-47e8-bb99-c6bcd5939ab0\") " pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn" Nov 03 20:38:37.230659 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:37.230599 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjpv5\" (UniqueName: \"kubernetes.io/projected/feae879c-5eba-47e8-bb99-c6bcd5939ab0-kube-api-access-pjpv5\") pod \"maas-api-5ff6ff6f8b-2blhn\" (UID: \"feae879c-5eba-47e8-bb99-c6bcd5939ab0\") " pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn" Nov 03 20:38:37.285393 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:37.285366 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn" Nov 03 20:38:37.689251 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:37.689211 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn"] Nov 03 20:38:37.696690 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:38:37.696663 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeae879c_5eba_47e8_bb99_c6bcd5939ab0.slice/crio-3cc0b42b4b0ebfc18d8e9cfb72c9639c7f1ceee83aa374ba307a94638ed4469d WatchSource:0}: Error finding container 3cc0b42b4b0ebfc18d8e9cfb72c9639c7f1ceee83aa374ba307a94638ed4469d: Status 404 returned error can't find the container with id 3cc0b42b4b0ebfc18d8e9cfb72c9639c7f1ceee83aa374ba307a94638ed4469d Nov 03 20:38:37.841286 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:37.841259 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn" event={"ID":"feae879c-5eba-47e8-bb99-c6bcd5939ab0","Type":"ContainerStarted","Data":"3cc0b42b4b0ebfc18d8e9cfb72c9639c7f1ceee83aa374ba307a94638ed4469d"} Nov 03 20:38:40.858546 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:40.858504 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn" event={"ID":"feae879c-5eba-47e8-bb99-c6bcd5939ab0","Type":"ContainerStarted","Data":"8e3642a58aef8d98af5d1a2e305434b12ace3e82341f0c1cba93626bd624cda4"} Nov 03 20:38:40.858965 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:40.858618 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn" Nov 03 20:38:40.966581 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:40.966535 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn" podStartSLOduration=2.153499246 podStartE2EDuration="4.966522279s" totalImagesPullingTime="2.813023033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:40.965718213 +0000 UTC m=+592.523708735" watchObservedRunningTime="2026-11-03 20:38:40.966522279 +0000 UTC m=+592.524512777" Nov 03 20:38:46.868147 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:46.868120 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn" Nov 03 20:38:48.592684 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:48.592657 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 20:38:48.592684 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:38:48.592677 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 20:38:51.743320 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:38:51.743282 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:39:21.743607 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:39:21.743566 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:39:48.468101 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:48.468070 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64"] Nov 03 20:39:48.471633 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:48.471616 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" Nov 03 20:39:48.570417 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:48.570391 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 20:39:48.570417 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:48.570394 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 20:39:48.570627 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:48.570502 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mr542\"" Nov 03 20:39:48.570627 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:48.570525 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 03 20:39:48.572022 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:48.572003 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64"] Nov 03 20:39:48.573992 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:48.573975 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e855d53-7358-4981-8313-16b0dbf17475-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vkr64\" (UID: \"8e855d53-7358-4981-8313-16b0dbf17475\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" Nov 03 20:39:48.574041 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:48.574004 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e855d53-7358-4981-8313-16b0dbf17475-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vkr64\" (UID: \"8e855d53-7358-4981-8313-16b0dbf17475\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" Nov 03 20:39:48.574074 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:48.574044 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e855d53-7358-4981-8313-16b0dbf17475-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vkr64\" (UID: \"8e855d53-7358-4981-8313-16b0dbf17475\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" Nov 03 20:39:48.574118 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:48.574107 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e855d53-7358-4981-8313-16b0dbf17475-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vkr64\" (UID: \"8e855d53-7358-4981-8313-16b0dbf17475\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" Nov 03 20:39:48.574186 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:48.574171 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e855d53-7358-4981-8313-16b0dbf17475-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vkr64\" (UID: \"8e855d53-7358-4981-8313-16b0dbf17475\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" Nov 03 20:39:48.574227 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:48.574212 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcjdv\" (UniqueName: \"kubernetes.io/projected/8e855d53-7358-4981-8313-16b0dbf17475-kube-api-access-qcjdv\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vkr64\" (UID: \"8e855d53-7358-4981-8313-16b0dbf17475\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" Nov 03 20:39:48.675020 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:48.674993 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e855d53-7358-4981-8313-16b0dbf17475-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vkr64\" (UID: \"8e855d53-7358-4981-8313-16b0dbf17475\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" Nov 03 20:39:48.675020 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:48.675026 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e855d53-7358-4981-8313-16b0dbf17475-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vkr64\" (UID: \"8e855d53-7358-4981-8313-16b0dbf17475\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" Nov 03 20:39:48.675579 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:48.675324 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e855d53-7358-4981-8313-16b0dbf17475-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vkr64\" (UID: \"8e855d53-7358-4981-8313-16b0dbf17475\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" Nov 03 20:39:48.675579 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:48.675387 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcjdv\" (UniqueName: \"kubernetes.io/projected/8e855d53-7358-4981-8313-16b0dbf17475-kube-api-access-qcjdv\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vkr64\" (UID: \"8e855d53-7358-4981-8313-16b0dbf17475\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" Nov 03 20:39:48.675579 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:48.675455 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e855d53-7358-4981-8313-16b0dbf17475-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vkr64\" (UID: \"8e855d53-7358-4981-8313-16b0dbf17475\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" Nov 03 20:39:48.675579 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:48.675458 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e855d53-7358-4981-8313-16b0dbf17475-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vkr64\" (UID: \"8e855d53-7358-4981-8313-16b0dbf17475\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" Nov 03 20:39:48.675579 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:48.675506 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e855d53-7358-4981-8313-16b0dbf17475-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vkr64\" (UID: \"8e855d53-7358-4981-8313-16b0dbf17475\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" Nov 03 20:39:48.675847 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:48.675723 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e855d53-7358-4981-8313-16b0dbf17475-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vkr64\" (UID: \"8e855d53-7358-4981-8313-16b0dbf17475\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" Nov 03 20:39:48.675847 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:48.675758 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e855d53-7358-4981-8313-16b0dbf17475-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vkr64\" (UID: \"8e855d53-7358-4981-8313-16b0dbf17475\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" Nov 03 20:39:48.677512 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:48.677494 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e855d53-7358-4981-8313-16b0dbf17475-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vkr64\" (UID: \"8e855d53-7358-4981-8313-16b0dbf17475\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" Nov 03 20:39:48.677566 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:48.677539 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e855d53-7358-4981-8313-16b0dbf17475-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vkr64\" (UID: \"8e855d53-7358-4981-8313-16b0dbf17475\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" Nov 03 20:39:48.728630 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:48.728578 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcjdv\" (UniqueName: \"kubernetes.io/projected/8e855d53-7358-4981-8313-16b0dbf17475-kube-api-access-qcjdv\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vkr64\" (UID: \"8e855d53-7358-4981-8313-16b0dbf17475\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" Nov 03 20:39:48.784390 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:48.784361 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" Nov 03 20:39:49.154183 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:49.154150 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64"] Nov 03 20:39:49.160326 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:39:49.160302 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e855d53_7358_4981_8313_16b0dbf17475.slice/crio-f9b91aa0abdc8e9a7ccb28c0c72db2e02618825a666a3ef194805814bdcee0be WatchSource:0}: Error finding container f9b91aa0abdc8e9a7ccb28c0c72db2e02618825a666a3ef194805814bdcee0be: Status 404 returned error can't find the container with id f9b91aa0abdc8e9a7ccb28c0c72db2e02618825a666a3ef194805814bdcee0be Nov 03 20:39:50.155088 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:50.155051 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" event={"ID":"8e855d53-7358-4981-8313-16b0dbf17475","Type":"ContainerStarted","Data":"f9b91aa0abdc8e9a7ccb28c0c72db2e02618825a666a3ef194805814bdcee0be"} Nov 03 20:39:51.164703 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:51.164664 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" event={"ID":"8e855d53-7358-4981-8313-16b0dbf17475","Type":"ContainerStarted","Data":"cea9905b50fc28c2b768c6bcff2e0e3b8c05e35ffed6d9fd771ddb88553ce79c"} Nov 03 20:39:51.273696 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:51.273644 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" podStartSLOduration=1.7218792440000001 podStartE2EDuration="3.273630579s" totalImagesPullingTime="1.551751335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:51.271955594 +0000 UTC m=+662.829946120" watchObservedRunningTime="2026-11-03 20:39:51.273630579 +0000 UTC m=+662.831621077" Nov 03 20:39:51.743105 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:39:51.743071 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:39:51.785421 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:51.785397 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" Nov 03 20:39:55.937896 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:55.937862 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-4bmw5"] Nov 03 20:39:55.941514 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:55.941494 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" Nov 03 20:39:56.039215 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:56.039186 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e6fd928-16a6-4624-b8a1-8e3027433637-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-4bmw5\" (UID: \"8e6fd928-16a6-4624-b8a1-8e3027433637\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" Nov 03 20:39:56.039215 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:56.039215 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6fd928-16a6-4624-b8a1-8e3027433637-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-4bmw5\" (UID: \"8e6fd928-16a6-4624-b8a1-8e3027433637\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" Nov 03 20:39:56.039421 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:56.039245 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhsv4\" (UniqueName: \"kubernetes.io/projected/8e6fd928-16a6-4624-b8a1-8e3027433637-kube-api-access-bhsv4\") pod \"echo-simulator-kserve-64b6f8f96c-4bmw5\" (UID: \"8e6fd928-16a6-4624-b8a1-8e3027433637\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" Nov 03 20:39:56.039421 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:56.039381 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e6fd928-16a6-4624-b8a1-8e3027433637-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-4bmw5\" (UID: \"8e6fd928-16a6-4624-b8a1-8e3027433637\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" Nov 03 20:39:56.039421 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:56.039406 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e6fd928-16a6-4624-b8a1-8e3027433637-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-4bmw5\" (UID: \"8e6fd928-16a6-4624-b8a1-8e3027433637\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" Nov 03 20:39:56.039521 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:56.039426 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e6fd928-16a6-4624-b8a1-8e3027433637-home\") pod \"echo-simulator-kserve-64b6f8f96c-4bmw5\" (UID: \"8e6fd928-16a6-4624-b8a1-8e3027433637\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" Nov 03 20:39:56.040222 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:56.040207 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 20:39:56.041779 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:56.041760 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-4bmw5"] Nov 03 20:39:56.140547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:56.140508 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e6fd928-16a6-4624-b8a1-8e3027433637-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-4bmw5\" (UID: \"8e6fd928-16a6-4624-b8a1-8e3027433637\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" Nov 03 20:39:56.140547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:56.140545 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6fd928-16a6-4624-b8a1-8e3027433637-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-4bmw5\" (UID: \"8e6fd928-16a6-4624-b8a1-8e3027433637\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" Nov 03 20:39:56.140786 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:56.140582 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhsv4\" (UniqueName: \"kubernetes.io/projected/8e6fd928-16a6-4624-b8a1-8e3027433637-kube-api-access-bhsv4\") pod \"echo-simulator-kserve-64b6f8f96c-4bmw5\" (UID: \"8e6fd928-16a6-4624-b8a1-8e3027433637\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" Nov 03 20:39:56.140786 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:56.140722 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e6fd928-16a6-4624-b8a1-8e3027433637-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-4bmw5\" (UID: \"8e6fd928-16a6-4624-b8a1-8e3027433637\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" Nov 03 20:39:56.140786 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:56.140760 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e6fd928-16a6-4624-b8a1-8e3027433637-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-4bmw5\" (UID: \"8e6fd928-16a6-4624-b8a1-8e3027433637\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" Nov 03 20:39:56.140786 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:56.140783 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e6fd928-16a6-4624-b8a1-8e3027433637-home\") pod \"echo-simulator-kserve-64b6f8f96c-4bmw5\" (UID: \"8e6fd928-16a6-4624-b8a1-8e3027433637\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" Nov 03 20:39:56.141038 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:56.140879 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e6fd928-16a6-4624-b8a1-8e3027433637-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-4bmw5\" (UID: \"8e6fd928-16a6-4624-b8a1-8e3027433637\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" Nov 03 20:39:56.141102 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:56.141083 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e6fd928-16a6-4624-b8a1-8e3027433637-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-4bmw5\" (UID: \"8e6fd928-16a6-4624-b8a1-8e3027433637\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" Nov 03 20:39:56.141150 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:56.141126 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e6fd928-16a6-4624-b8a1-8e3027433637-home\") pod \"echo-simulator-kserve-64b6f8f96c-4bmw5\" (UID: \"8e6fd928-16a6-4624-b8a1-8e3027433637\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" Nov 03 20:39:56.142934 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:56.142912 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e6fd928-16a6-4624-b8a1-8e3027433637-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-4bmw5\" (UID: \"8e6fd928-16a6-4624-b8a1-8e3027433637\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" Nov 03 20:39:56.143225 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:56.143207 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6fd928-16a6-4624-b8a1-8e3027433637-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-4bmw5\" (UID: \"8e6fd928-16a6-4624-b8a1-8e3027433637\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" Nov 03 20:39:56.194942 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:56.194891 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhsv4\" (UniqueName: \"kubernetes.io/projected/8e6fd928-16a6-4624-b8a1-8e3027433637-kube-api-access-bhsv4\") pod \"echo-simulator-kserve-64b6f8f96c-4bmw5\" (UID: \"8e6fd928-16a6-4624-b8a1-8e3027433637\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" Nov 03 20:39:56.254041 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:56.254010 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" Nov 03 20:39:56.620921 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:56.620883 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-4bmw5"] Nov 03 20:39:56.627293 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:39:56.627265 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6fd928_16a6_4624_b8a1_8e3027433637.slice/crio-76174e356b0fe4dcca35f0ebc645c15d18c53ceb5c7cc56c696d84bc98d0100c WatchSource:0}: Error finding container 76174e356b0fe4dcca35f0ebc645c15d18c53ceb5c7cc56c696d84bc98d0100c: Status 404 returned error can't find the container with id 76174e356b0fe4dcca35f0ebc645c15d18c53ceb5c7cc56c696d84bc98d0100c Nov 03 20:39:57.191047 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:57.191008 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" event={"ID":"8e6fd928-16a6-4624-b8a1-8e3027433637","Type":"ContainerStarted","Data":"76174e356b0fe4dcca35f0ebc645c15d18c53ceb5c7cc56c696d84bc98d0100c"} Nov 03 20:39:58.784690 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:58.784616 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" Nov 03 20:39:58.798409 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:58.798385 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" Nov 03 20:39:59.201917 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:59.201876 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" event={"ID":"8e6fd928-16a6-4624-b8a1-8e3027433637","Type":"ContainerStarted","Data":"ae1f2e0bf7a3a5ece9d8fd201cec4f7cd2c227c7c474ecf06fa7cf8486420567"} Nov 03 20:39:59.213621 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:59.213597 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vkr64" Nov 03 20:39:59.254416 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:59.254383 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" Nov 03 20:39:59.311070 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:39:59.311015 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" podStartSLOduration=2.426073957 podStartE2EDuration="4.311001398s" totalImagesPullingTime="1.884927441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:59.309890763 +0000 UTC m=+670.867881262" watchObservedRunningTime="2026-11-03 20:39:59.311001398 +0000 UTC m=+670.868991897" Nov 03 20:40:02.855202 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:02.855166 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4"] Nov 03 20:40:02.858575 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:02.858555 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:02.907569 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:02.907539 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 20:40:02.954586 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:02.954553 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-spnvt\"" Nov 03 20:40:02.956235 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:02.956214 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4"] Nov 03 20:40:03.002900 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.002867 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-765j9\" (UniqueName: \"kubernetes.io/projected/d07d8d19-5ed5-4f0b-9084-390494a695c6-kube-api-access-765j9\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.003061 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.002917 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.003061 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.002940 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.003061 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.002982 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d07d8d19-5ed5-4f0b-9084-390494a695c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.003201 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.003083 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.003201 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.003180 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.003293 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.003216 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.003293 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.003234 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.003293 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.003269 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.104065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.104038 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.104065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.104065 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d07d8d19-5ed5-4f0b-9084-390494a695c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.104237 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.104083 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.104237 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.104123 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.104237 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.104161 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.104237 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.104182 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.104237 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.104208 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.104584 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.104244 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-765j9\" (UniqueName: \"kubernetes.io/projected/d07d8d19-5ed5-4f0b-9084-390494a695c6-kube-api-access-765j9\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.104584 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.104305 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.104719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.104633 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.104719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.104661 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.104833 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.104753 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.104833 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.104795 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.105263 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.105211 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d07d8d19-5ed5-4f0b-9084-390494a695c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.106600 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.106578 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.106720 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.106702 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.157866 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.157838 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.158076 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.158058 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-765j9\" (UniqueName: \"kubernetes.io/projected/d07d8d19-5ed5-4f0b-9084-390494a695c6-kube-api-access-765j9\") pod \"maas-default-gateway-openshift-default-696f47c6c-dbbg4\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.170861 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.170839 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:03.594079 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.594046 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4"] Nov 03 20:40:03.640652 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:40:03.640621 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd07d8d19_5ed5_4f0b_9084_390494a695c6.slice/crio-a87cabaf3bc2f13cfe2175b53469a60332645f7e20deea2a13f98c611e584427 WatchSource:0}: Error finding container a87cabaf3bc2f13cfe2175b53469a60332645f7e20deea2a13f98c611e584427: Status 404 returned error can't find the container with id a87cabaf3bc2f13cfe2175b53469a60332645f7e20deea2a13f98c611e584427 Nov 03 20:40:03.642556 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:03.642537 4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:40:04.227114 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:04.227078 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" event={"ID":"d07d8d19-5ed5-4f0b-9084-390494a695c6","Type":"ContainerStarted","Data":"a87cabaf3bc2f13cfe2175b53469a60332645f7e20deea2a13f98c611e584427"} Nov 03 20:40:05.536750 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:05.536714 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856"] Nov 03 20:40:05.541319 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:05.541301 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" Nov 03 20:40:05.594449 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:05.594416 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 03 20:40:05.625875 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:05.625835 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56z4v\" (UniqueName: \"kubernetes.io/projected/32fb3160-1d88-4cf6-ac03-4e2f030414cc-kube-api-access-56z4v\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw856\" (UID: \"32fb3160-1d88-4cf6-ac03-4e2f030414cc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" Nov 03 20:40:05.626023 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:05.625881 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32fb3160-1d88-4cf6-ac03-4e2f030414cc-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw856\" (UID: \"32fb3160-1d88-4cf6-ac03-4e2f030414cc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" Nov 03 20:40:05.626023 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:05.625908 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32fb3160-1d88-4cf6-ac03-4e2f030414cc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw856\" (UID: \"32fb3160-1d88-4cf6-ac03-4e2f030414cc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" Nov 03 20:40:05.626152 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:05.626019 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32fb3160-1d88-4cf6-ac03-4e2f030414cc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw856\" (UID: \"32fb3160-1d88-4cf6-ac03-4e2f030414cc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" Nov 03 20:40:05.626152 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:05.626083 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32fb3160-1d88-4cf6-ac03-4e2f030414cc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw856\" (UID: \"32fb3160-1d88-4cf6-ac03-4e2f030414cc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" Nov 03 20:40:05.626242 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:05.626169 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32fb3160-1d88-4cf6-ac03-4e2f030414cc-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw856\" (UID: \"32fb3160-1d88-4cf6-ac03-4e2f030414cc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" Nov 03 20:40:05.642216 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:05.642192 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856"] Nov 03 20:40:05.726788 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:05.726752 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56z4v\" (UniqueName: \"kubernetes.io/projected/32fb3160-1d88-4cf6-ac03-4e2f030414cc-kube-api-access-56z4v\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw856\" (UID: \"32fb3160-1d88-4cf6-ac03-4e2f030414cc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" Nov 03 20:40:05.726953 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:05.726796 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32fb3160-1d88-4cf6-ac03-4e2f030414cc-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw856\" (UID: \"32fb3160-1d88-4cf6-ac03-4e2f030414cc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" Nov 03 20:40:05.726953 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:05.726812 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32fb3160-1d88-4cf6-ac03-4e2f030414cc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw856\" (UID: \"32fb3160-1d88-4cf6-ac03-4e2f030414cc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" Nov 03 20:40:05.726953 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:05.726852 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32fb3160-1d88-4cf6-ac03-4e2f030414cc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw856\" (UID: \"32fb3160-1d88-4cf6-ac03-4e2f030414cc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" Nov 03 20:40:05.726953 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:05.726895 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32fb3160-1d88-4cf6-ac03-4e2f030414cc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw856\" (UID: \"32fb3160-1d88-4cf6-ac03-4e2f030414cc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" Nov 03 20:40:05.726953 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:05.726923 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32fb3160-1d88-4cf6-ac03-4e2f030414cc-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw856\" (UID: \"32fb3160-1d88-4cf6-ac03-4e2f030414cc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" Nov 03 20:40:05.727306 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:05.727284 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32fb3160-1d88-4cf6-ac03-4e2f030414cc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw856\" (UID: \"32fb3160-1d88-4cf6-ac03-4e2f030414cc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" Nov 03 20:40:05.727418 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:05.727318 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32fb3160-1d88-4cf6-ac03-4e2f030414cc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw856\" (UID: \"32fb3160-1d88-4cf6-ac03-4e2f030414cc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" Nov 03 20:40:05.727418 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:05.727370 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32fb3160-1d88-4cf6-ac03-4e2f030414cc-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw856\" (UID: \"32fb3160-1d88-4cf6-ac03-4e2f030414cc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" Nov 03 20:40:05.729190 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:05.729172 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32fb3160-1d88-4cf6-ac03-4e2f030414cc-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw856\" (UID: \"32fb3160-1d88-4cf6-ac03-4e2f030414cc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" Nov 03 20:40:05.729487 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:05.729464 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32fb3160-1d88-4cf6-ac03-4e2f030414cc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw856\" (UID: \"32fb3160-1d88-4cf6-ac03-4e2f030414cc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" Nov 03 20:40:05.782064 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:05.782036 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56z4v\" (UniqueName: \"kubernetes.io/projected/32fb3160-1d88-4cf6-ac03-4e2f030414cc-kube-api-access-56z4v\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw856\" (UID: \"32fb3160-1d88-4cf6-ac03-4e2f030414cc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" Nov 03 20:40:05.853866 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:05.853838 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" Nov 03 20:40:06.230067 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:06.229993 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856"] Nov 03 20:40:06.234805 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:40:06.234771 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32fb3160_1d88_4cf6_ac03_4e2f030414cc.slice/crio-f0437a8bda5e18d6f4531f677a7cc37dab47f594da3c3a48a8e01d0a230aee71 WatchSource:0}: Error finding container f0437a8bda5e18d6f4531f677a7cc37dab47f594da3c3a48a8e01d0a230aee71: Status 404 returned error can't find the container with id f0437a8bda5e18d6f4531f677a7cc37dab47f594da3c3a48a8e01d0a230aee71 Nov 03 20:40:06.254548 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:06.254522 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" Nov 03 20:40:06.273294 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:06.273035 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" Nov 03 20:40:07.244720 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:07.244617 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" event={"ID":"32fb3160-1d88-4cf6-ac03-4e2f030414cc","Type":"ContainerStarted","Data":"54f1ed29bf02731f3f5f75fb813c52ba101cd2068a7edafc249140c13c9589b6"} Nov 03 20:40:07.244720 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:07.244699 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" event={"ID":"32fb3160-1d88-4cf6-ac03-4e2f030414cc","Type":"ContainerStarted","Data":"f0437a8bda5e18d6f4531f677a7cc37dab47f594da3c3a48a8e01d0a230aee71"} Nov 03 20:40:07.262587 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:07.262559 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-4bmw5" Nov 03 20:40:07.353527 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:07.353473 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" podStartSLOduration=1.7359169410000002 podStartE2EDuration="2.353459685s" totalImagesPullingTime="617.542744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:07.352632813 +0000 UTC m=+678.910623336" watchObservedRunningTime="2026-11-03 20:40:07.353459685 +0000 UTC m=+678.911450205" Nov 03 20:40:07.854670 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:07.854631 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" Nov 03 20:40:07.973207 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:07.973176 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:40:07.973299 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:07.973243 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:40:07.973299 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:07.973277 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:40:08.252628 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:08.252549 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" event={"ID":"d07d8d19-5ed5-4f0b-9084-390494a695c6","Type":"ContainerStarted","Data":"d03028ce68d694afe8f787a42a1b19eb91ab15cdc8462cba5e3cd7a729ba77a7"} Nov 03 20:40:08.362938 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:08.362886 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" podStartSLOduration=2.033192233 podStartE2EDuration="6.362871152s" totalImagesPullingTime="4.329678919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:08.361708288 +0000 UTC m=+679.919698788" watchObservedRunningTime="2026-11-03 20:40:08.362871152 +0000 UTC m=+679.920861651" Nov 03 20:40:09.171583 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:09.171549 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:09.173105 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:09.173080 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-dbbg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 03 20:40:09.173225 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:09.173133 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" podUID="d07d8d19-5ed5-4f0b-9084-390494a695c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 03 20:40:10.171277 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:10.171235 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-dbbg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 03 20:40:10.171725 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:10.171307 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" podUID="d07d8d19-5ed5-4f0b-9084-390494a695c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 03 20:40:11.171447 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:11.171409 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-dbbg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 03 20:40:11.171828 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:11.171483 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" podUID="d07d8d19-5ed5-4f0b-9084-390494a695c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 03 20:40:12.171997 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:12.171964 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-dbbg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 03 20:40:12.172378 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:12.172026 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" podUID="d07d8d19-5ed5-4f0b-9084-390494a695c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 03 20:40:13.171615 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:13.171574 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-dbbg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 03 20:40:13.171833 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:13.171654 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" podUID="d07d8d19-5ed5-4f0b-9084-390494a695c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 03 20:40:14.171610 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:14.171567 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-dbbg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 03 20:40:14.172102 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:14.171646 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" podUID="d07d8d19-5ed5-4f0b-9084-390494a695c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 03 20:40:15.854081 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:15.854041 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" Nov 03 20:40:15.866801 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:15.866774 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" Nov 03 20:40:16.172497 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:16.172420 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-dbbg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:40:16.172497 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:16.172473 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" podUID="d07d8d19-5ed5-4f0b-9084-390494a695c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:40:16.299181 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:16.299150 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw856" Nov 03 20:40:17.213535 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:17.213480 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-dbbg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:40:17.214033 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:17.213554 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" podUID="d07d8d19-5ed5-4f0b-9084-390494a695c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:40:17.648217 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:17.648184 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:17.648578 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:17.648557 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:17.649374 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:17.649353 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:40:21.743513 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:40:21.743475 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:40:33.084367 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.084316 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l"] Nov 03 20:40:33.088030 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.088011 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.185993 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.185964 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l"] Nov 03 20:40:33.195212 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.195185 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.195437 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.195269 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.195545 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.195423 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzwh9\" (UniqueName: \"kubernetes.io/projected/a3f1f367-79f6-4068-90bc-6373d5651ab4-kube-api-access-fzwh9\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.195545 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.195467 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f1f367-79f6-4068-90bc-6373d5651ab4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.195662 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.195598 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.195710 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.195659 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.195710 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.195698 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.195963 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.195725 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.195963 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.195757 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.296665 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.296633 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f1f367-79f6-4068-90bc-6373d5651ab4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.296823 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.296682 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.296823 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.296709 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.296823 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.296738 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.296823 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.296761 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.296823 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.296788 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.297083 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.296850 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.297083 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.296922 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.297083 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.296981 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzwh9\" (UniqueName: \"kubernetes.io/projected/a3f1f367-79f6-4068-90bc-6373d5651ab4-kube-api-access-fzwh9\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.297294 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.297139 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.297294 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.297254 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.297294 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.297253 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.297501 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.297476 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f1f367-79f6-4068-90bc-6373d5651ab4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.297561 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.297545 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.299545 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.299521 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.299669 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.299527 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.350758 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.350683 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.350919 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.350856 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzwh9\" (UniqueName: \"kubernetes.io/projected/a3f1f367-79f6-4068-90bc-6373d5651ab4-kube-api-access-fzwh9\") pod \"maas-default-gateway-openshift-default-696f47c6c-9pl8l\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.400562 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.400528 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:33.825101 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.825067 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l"] Nov 03 20:40:33.831002 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:40:33.830936 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f1f367_79f6_4068_90bc_6373d5651ab4.slice/crio-60f169b62aae21b58c1656d8f42bb5eef8f43b6e78e8d350dc2f39035b5dc1f0 WatchSource:0}: Error finding container 60f169b62aae21b58c1656d8f42bb5eef8f43b6e78e8d350dc2f39035b5dc1f0: Status 404 returned error can't find the container with id 60f169b62aae21b58c1656d8f42bb5eef8f43b6e78e8d350dc2f39035b5dc1f0 Nov 03 20:40:33.833500 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.833470 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:40:33.833585 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.833536 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:40:33.833585 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:33.833564 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:40:34.371805 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:34.371760 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" event={"ID":"a3f1f367-79f6-4068-90bc-6373d5651ab4","Type":"ContainerStarted","Data":"a7995bb6789c04d6541cbcc96ca030fec43b65b830e669181976a0716a2a24a3"} Nov 03 20:40:34.372199 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:34.371853 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" event={"ID":"a3f1f367-79f6-4068-90bc-6373d5651ab4","Type":"ContainerStarted","Data":"60f169b62aae21b58c1656d8f42bb5eef8f43b6e78e8d350dc2f39035b5dc1f0"} Nov 03 20:40:34.406360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:34.405994 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:34.406360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:34.406137 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-9pl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 03 20:40:34.406360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:34.406189 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" podUID="a3f1f367-79f6-4068-90bc-6373d5651ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 03 20:40:34.482937 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:34.482884 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" podStartSLOduration=1.482872395 podStartE2EDuration="1.482872395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:34.482058949 +0000 UTC m=+706.040049469" watchObservedRunningTime="2026-11-03 20:40:34.482872395 +0000 UTC m=+706.040862895" Nov 03 20:40:35.408767 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:35.408733 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-9pl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 03 20:40:35.409254 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:35.408815 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" podUID="a3f1f367-79f6-4068-90bc-6373d5651ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 03 20:40:36.405358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:36.403718 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-9pl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 03 20:40:36.405358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:36.403773 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" podUID="a3f1f367-79f6-4068-90bc-6373d5651ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 03 20:40:37.408855 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:37.407916 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-9pl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 03 20:40:37.408855 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:37.407973 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" podUID="a3f1f367-79f6-4068-90bc-6373d5651ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 03 20:40:38.406360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:38.405702 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-9pl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 03 20:40:38.406360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:38.405762 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" podUID="a3f1f367-79f6-4068-90bc-6373d5651ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 03 20:40:39.403233 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:39.402746 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-9pl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 03 20:40:39.403233 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:39.402831 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" podUID="a3f1f367-79f6-4068-90bc-6373d5651ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 03 20:40:40.401730 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:40.401689 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-9pl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 03 20:40:40.401932 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:40.401752 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" podUID="a3f1f367-79f6-4068-90bc-6373d5651ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 03 20:40:42.026727 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:42.026681 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:42.027220 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:42.027199 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:42.027912 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:40:42.027894 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:40:51.743323 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:40:51.743287 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:41:18.408210 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.408168 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn"] Nov 03 20:41:18.412041 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.412019 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.504984 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.504949 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.505137 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.504998 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.505137 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.505015 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.505137 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.505100 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.505238 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.505135 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a7db53-d5c6-4339-81c4-38c917ecaa15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.505238 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.505161 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vq8f\" (UniqueName: \"kubernetes.io/projected/16a7db53-d5c6-4339-81c4-38c917ecaa15-kube-api-access-5vq8f\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.505303 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.505253 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.505303 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.505287 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.505402 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.505305 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.510653 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.510625 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn"] Nov 03 20:41:18.605782 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.605749 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.605944 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.605803 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.605944 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.605828 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.605944 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.605863 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.606084 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.606011 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a7db53-d5c6-4339-81c4-38c917ecaa15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.606084 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.606052 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vq8f\" (UniqueName: \"kubernetes.io/projected/16a7db53-d5c6-4339-81c4-38c917ecaa15-kube-api-access-5vq8f\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.606164 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.606115 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.606209 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.606173 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.606209 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.606176 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.606209 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.606204 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.606499 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.606458 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.606604 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.606555 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.606726 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.606703 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a7db53-d5c6-4339-81c4-38c917ecaa15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.606801 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.606745 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.608342 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.608313 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.608572 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.608555 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.659994 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.659926 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.660621 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.660597 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vq8f\" (UniqueName: \"kubernetes.io/projected/16a7db53-d5c6-4339-81c4-38c917ecaa15-kube-api-access-5vq8f\") pod \"maas-default-gateway-openshift-default-696f47c6c-bhrgn\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:18.724362 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:18.724326 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:19.143854 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:19.143821 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn"] Nov 03 20:41:19.150561 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:41:19.150536 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a7db53_d5c6_4339_81c4_38c917ecaa15.slice/crio-f247a19bef95e180c529f3ae38b629428002a552274a6aa4f7122459f60bec3a WatchSource:0}: Error finding container f247a19bef95e180c529f3ae38b629428002a552274a6aa4f7122459f60bec3a: Status 404 returned error can't find the container with id f247a19bef95e180c529f3ae38b629428002a552274a6aa4f7122459f60bec3a Nov 03 20:41:19.152856 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:19.152826 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:41:19.152929 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:19.152895 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:41:19.152929 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:19.152921 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:41:19.612174 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:19.611141 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" event={"ID":"16a7db53-d5c6-4339-81c4-38c917ecaa15","Type":"ContainerStarted","Data":"25d35cf4d867d9bec71cf2012596a660910a36bef4cb8a53f0f2a7b67a88012f"} Nov 03 20:41:19.612174 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:19.611215 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" event={"ID":"16a7db53-d5c6-4339-81c4-38c917ecaa15","Type":"ContainerStarted","Data":"f247a19bef95e180c529f3ae38b629428002a552274a6aa4f7122459f60bec3a"} Nov 03 20:41:19.724941 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:19.724912 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:19.757415 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:19.757368 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" podStartSLOduration=1.757351509 podStartE2EDuration="1.757351509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:19.721576963 +0000 UTC m=+751.279567482" watchObservedRunningTime="2026-11-03 20:41:19.757351509 +0000 UTC m=+751.315342007" Nov 03 20:41:20.725987 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:20.725956 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-bhrgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 03 20:41:20.726395 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:20.726011 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" podUID="16a7db53-d5c6-4339-81c4-38c917ecaa15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 03 20:41:21.724754 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:21.724722 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-bhrgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 03 20:41:21.724918 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:21.724783 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" podUID="16a7db53-d5c6-4339-81c4-38c917ecaa15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 03 20:41:21.743908 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:41:21.743875 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:41:22.725488 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:22.725450 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-bhrgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 03 20:41:22.725697 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:22.725513 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" podUID="16a7db53-d5c6-4339-81c4-38c917ecaa15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 03 20:41:23.725578 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:23.725545 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-bhrgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 03 20:41:23.726037 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:23.725595 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" podUID="16a7db53-d5c6-4339-81c4-38c917ecaa15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 03 20:41:24.725050 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:24.725016 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-bhrgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 03 20:41:24.725244 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:24.725112 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" podUID="16a7db53-d5c6-4339-81c4-38c917ecaa15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 03 20:41:25.725788 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:25.725747 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-bhrgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 03 20:41:25.726283 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:25.725828 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" podUID="16a7db53-d5c6-4339-81c4-38c917ecaa15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 03 20:41:27.380160 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:27.380128 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:27.380671 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:27.380443 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:27.381272 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:41:27.381252 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:41:51.744036 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:41:51.744001 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:42:06.684650 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.684617 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7"] Nov 03 20:42:06.688116 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.688097 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:06.737379 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.737351 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-1ec892-openshift-default-dockercfg-dds29\"" Nov 03 20:42:06.786664 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.786632 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7"] Nov 03 20:42:06.851752 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.851724 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:06.851910 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.851753 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151eca7e-97d2-4220-a630-71e7df7d9f24-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:06.851910 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.851789 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:06.851910 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.851856 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:06.852063 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.851932 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:06.852063 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.851976 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:06.852063 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.852023 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vf25\" (UniqueName: \"kubernetes.io/projected/151eca7e-97d2-4220-a630-71e7df7d9f24-kube-api-access-6vf25\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:06.852156 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.852077 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:06.852156 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.852114 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:06.953424 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.953320 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:06.953424 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.953411 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:06.953656 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.953438 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151eca7e-97d2-4220-a630-71e7df7d9f24-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:06.953656 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.953467 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:06.953656 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.953597 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:06.953656 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.953643 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:06.953866 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.953678 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:06.953866 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.953753 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:06.953866 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.953759 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vf25\" (UniqueName: \"kubernetes.io/projected/151eca7e-97d2-4220-a630-71e7df7d9f24-kube-api-access-6vf25\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:06.953866 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.953828 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:06.954064 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.953949 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:06.954064 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.954042 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:06.954183 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.954069 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:06.954458 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.954435 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151eca7e-97d2-4220-a630-71e7df7d9f24-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:06.956082 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.956028 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:06.956483 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.956365 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:06.961937 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.961912 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4"] Nov 03 20:42:06.966782 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:06.966762 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.007183 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.007159 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:07.007377 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.007357 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vf25\" (UniqueName: \"kubernetes.io/projected/151eca7e-97d2-4220-a630-71e7df7d9f24-kube-api-access-6vf25\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:07.019162 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.019133 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-b119f2-openshift-default-dockercfg-d7fgm\"" Nov 03 20:42:07.021914 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.021888 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4"] Nov 03 20:42:07.155082 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.155050 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.155082 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.155086 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.155270 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.155106 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.155270 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.155211 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.155270 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.155253 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.155400 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.155317 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjvxc\" (UniqueName: \"kubernetes.io/projected/16823f97-8616-45c4-8d13-e05b7d4bf432-kube-api-access-bjvxc\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.155437 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.155426 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.155470 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.155450 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.155507 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.155471 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16823f97-8616-45c4-8d13-e05b7d4bf432-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.256646 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.256575 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.256646 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.256605 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.256646 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.256624 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.256924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.256671 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.256924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.256710 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.256924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.256746 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjvxc\" (UniqueName: \"kubernetes.io/projected/16823f97-8616-45c4-8d13-e05b7d4bf432-kube-api-access-bjvxc\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.256924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.256816 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.256924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.256847 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.256924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.256872 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16823f97-8616-45c4-8d13-e05b7d4bf432-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.257199 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.257073 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.257199 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.257102 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.257199 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.257147 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.257296 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.257201 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.257576 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.257552 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16823f97-8616-45c4-8d13-e05b7d4bf432-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.259360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.259325 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.259428 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.259348 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.262858 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.262838 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr"] Nov 03 20:42:07.267349 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.267318 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.300265 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.300236 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:07.312611 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.312586 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.312970 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.312951 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjvxc\" (UniqueName: \"kubernetes.io/projected/16823f97-8616-45c4-8d13-e05b7d4bf432-kube-api-access-bjvxc\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.316800 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.316774 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-fd9afe-openshift-default-dockercfg-k948v\"" Nov 03 20:42:07.324053 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.324025 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr"] Nov 03 20:42:07.460000 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.459965 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.460181 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.460011 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.460181 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.460035 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3be0d37-42be-4b25-846b-d8538f590441-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.460181 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.460136 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.460350 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.460271 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3be0d37-42be-4b25-846b-d8538f590441-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.460390 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.460328 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl56q\" (UniqueName: \"kubernetes.io/projected/c3be0d37-42be-4b25-846b-d8538f590441-kube-api-access-zl56q\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.460390 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.460372 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.460478 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.460406 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3be0d37-42be-4b25-846b-d8538f590441-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.460514 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.460493 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.561327 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.561297 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3be0d37-42be-4b25-846b-d8538f590441-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.561537 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.561352 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl56q\" (UniqueName: \"kubernetes.io/projected/c3be0d37-42be-4b25-846b-d8538f590441-kube-api-access-zl56q\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.561537 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.561375 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.561537 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.561403 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3be0d37-42be-4b25-846b-d8538f590441-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.561537 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.561503 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.561537 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.561536 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.561803 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.561577 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.561803 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.561613 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3be0d37-42be-4b25-846b-d8538f590441-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.561803 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.561635 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.561950 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.561818 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.562090 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.562063 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.562191 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.562124 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.562252 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.562221 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.562526 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.562502 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3be0d37-42be-4b25-846b-d8538f590441-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.564224 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.564202 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.564353 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.564319 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3be0d37-42be-4b25-846b-d8538f590441-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.583252 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.583209 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:07.593934 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.593899 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g"] Nov 03 20:42:07.597849 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.597831 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.615462 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.615437 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl56q\" (UniqueName: \"kubernetes.io/projected/c3be0d37-42be-4b25-846b-d8538f590441-kube-api-access-zl56q\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.615598 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.615551 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3be0d37-42be-4b25-846b-d8538f590441-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.647076 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.647053 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-984cbc-openshift-default-dockercfg-qwh2c\"" Nov 03 20:42:07.655940 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.655911 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g"] Nov 03 20:42:07.662869 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.662842 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a092ea83-8438-4626-a3a6-0e076627ab11-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.662972 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.662952 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.663024 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.663010 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.663071 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.663053 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.663106 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.663095 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a092ea83-8438-4626-a3a6-0e076627ab11-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.663142 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.663116 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a092ea83-8438-4626-a3a6-0e076627ab11-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.663177 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.663159 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.663209 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.663178 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwmtp\" (UniqueName: \"kubernetes.io/projected/a092ea83-8438-4626-a3a6-0e076627ab11-kube-api-access-lwmtp\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.663241 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.663204 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.722221 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.722188 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7"] Nov 03 20:42:07.727886 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:42:07.727863 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151eca7e_97d2_4220_a630_71e7df7d9f24.slice/crio-8cb29f8f6b24fd1190bc6eb0f32d603b88768b16e7f11105f0762c9cdebfd1ab WatchSource:0}: Error finding container 8cb29f8f6b24fd1190bc6eb0f32d603b88768b16e7f11105f0762c9cdebfd1ab: Status 404 returned error can't find the container with id 8cb29f8f6b24fd1190bc6eb0f32d603b88768b16e7f11105f0762c9cdebfd1ab Nov 03 20:42:07.730096 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.730067 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:42:07.730150 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.730130 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:42:07.730183 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.730159 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:42:07.764213 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.764185 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.764383 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.764230 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.764383 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.764277 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.764383 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.764313 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a092ea83-8438-4626-a3a6-0e076627ab11-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.764555 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.764430 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a092ea83-8438-4626-a3a6-0e076627ab11-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.764555 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.764470 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.764555 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.764496 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwmtp\" (UniqueName: \"kubernetes.io/projected/a092ea83-8438-4626-a3a6-0e076627ab11-kube-api-access-lwmtp\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.764555 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.764529 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.764759 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.764582 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a092ea83-8438-4626-a3a6-0e076627ab11-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.764759 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.764600 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.764759 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.764660 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.764976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.764819 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.764976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.764937 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.765222 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.765201 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a092ea83-8438-4626-a3a6-0e076627ab11-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.766611 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.766595 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.766856 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.766840 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a092ea83-8438-4626-a3a6-0e076627ab11-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.818000 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.817941 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a092ea83-8438-4626-a3a6-0e076627ab11-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.818361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.818318 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwmtp\" (UniqueName: \"kubernetes.io/projected/a092ea83-8438-4626-a3a6-0e076627ab11-kube-api-access-lwmtp\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.840826 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.840793 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" event={"ID":"151eca7e-97d2-4220-a630-71e7df7d9f24","Type":"ContainerStarted","Data":"6bad799127269015d6095f4181642cbb66c6026e6391f9a53413074c11664262"} Nov 03 20:42:07.840956 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.840855 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" event={"ID":"151eca7e-97d2-4220-a630-71e7df7d9f24","Type":"ContainerStarted","Data":"8cb29f8f6b24fd1190bc6eb0f32d603b88768b16e7f11105f0762c9cdebfd1ab"} Nov 03 20:42:07.881080 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.881048 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:07.901059 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.901025 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc"] Nov 03 20:42:07.904805 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.904783 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:07.911328 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.911300 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:07.969953 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.955878 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" podStartSLOduration=1.955859554 podStartE2EDuration="1.955859554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:07.954553446 +0000 UTC m=+799.512543946" watchObservedRunningTime="2026-11-03 20:42:07.955859554 +0000 UTC m=+799.513850055" Nov 03 20:42:07.969953 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.962708 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc"] Nov 03 20:42:07.978375 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.977061 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4"] Nov 03 20:42:07.987240 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:42:07.987191 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16823f97_8616_45c4_8d13_e05b7d4bf432.slice/crio-cb36e66b3d63e7b8f94eed540320171e7d9afc85863c2c2793ff5bee2d4b1a39 WatchSource:0}: Error finding container cb36e66b3d63e7b8f94eed540320171e7d9afc85863c2c2793ff5bee2d4b1a39: Status 404 returned error can't find the container with id cb36e66b3d63e7b8f94eed540320171e7d9afc85863c2c2793ff5bee2d4b1a39 Nov 03 20:42:07.991842 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.991541 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:42:07.991842 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.991626 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:42:07.992489 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:07.992277 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:42:08.006358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.005953 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-a19980-openshift-default-dockercfg-nbrs7\"" Nov 03 20:42:08.075541 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.075510 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.075690 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.075568 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.075690 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.075595 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.075690 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.075671 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.075905 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.075884 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.075960 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.075929 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.076036 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.076015 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.076088 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.076064 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.076223 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.076208 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px8jt\" (UniqueName: \"kubernetes.io/projected/a3fb9508-37d2-47fa-bb02-ae7735c9f408-kube-api-access-px8jt\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.177006 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.176966 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.177195 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.177025 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.177195 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.177051 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.177195 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.177086 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.177195 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.177154 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.177195 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.177184 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.177535 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.177218 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.177535 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.177252 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.177535 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.177301 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px8jt\" (UniqueName: \"kubernetes.io/projected/a3fb9508-37d2-47fa-bb02-ae7735c9f408-kube-api-access-px8jt\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.177535 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.177443 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.177535 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.177503 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.177535 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.177512 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.177836 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.177814 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.178380 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.178356 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.180003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.179977 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.180123 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.180100 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.232376 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.232348 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.232880 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.232851 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px8jt\" (UniqueName: \"kubernetes.io/projected/a3fb9508-37d2-47fa-bb02-ae7735c9f408-kube-api-access-px8jt\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.279174 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.279136 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr"] Nov 03 20:42:08.283463 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:42:08.283431 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3be0d37_42be_4b25_846b_d8538f590441.slice/crio-55d263ee570a343e71e56d4c53600170a86a81541d5015ad0ae4c438f51c119c WatchSource:0}: Error finding container 55d263ee570a343e71e56d4c53600170a86a81541d5015ad0ae4c438f51c119c: Status 404 returned error can't find the container with id 55d263ee570a343e71e56d4c53600170a86a81541d5015ad0ae4c438f51c119c Nov 03 20:42:08.286230 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.286191 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:42:08.286316 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.286281 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:42:08.286396 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.286377 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:42:08.300884 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.300845 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:08.307485 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.307462 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:08.339197 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.339166 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g"] Nov 03 20:42:08.341872 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:42:08.341831 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda092ea83_8438_4626_a3a6_0e076627ab11.slice/crio-6e8535d840ddaf472387e12e37ffff68011daac3337b187c04ad5d7955cabc7d WatchSource:0}: Error finding container 6e8535d840ddaf472387e12e37ffff68011daac3337b187c04ad5d7955cabc7d: Status 404 returned error can't find the container with id 6e8535d840ddaf472387e12e37ffff68011daac3337b187c04ad5d7955cabc7d Nov 03 20:42:08.345088 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.345047 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:42:08.346406 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.345136 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:42:08.346406 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.345175 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:42:08.526887 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.526855 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:08.847403 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.847295 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" event={"ID":"c3be0d37-42be-4b25-846b-d8538f590441","Type":"ContainerStarted","Data":"cc23d43d708717c675b978b9771987a15b127af157120d844b8cbfb7b8120eea"} Nov 03 20:42:08.847403 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.847398 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" event={"ID":"c3be0d37-42be-4b25-846b-d8538f590441","Type":"ContainerStarted","Data":"55d263ee570a343e71e56d4c53600170a86a81541d5015ad0ae4c438f51c119c"} Nov 03 20:42:08.849289 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.849258 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" event={"ID":"16823f97-8616-45c4-8d13-e05b7d4bf432","Type":"ContainerStarted","Data":"c043fe533af5b8656516a7c85ade6f21fcc88348c5942b54e7aa8a3bb71674ce"} Nov 03 20:42:08.849462 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.849316 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" event={"ID":"16823f97-8616-45c4-8d13-e05b7d4bf432","Type":"ContainerStarted","Data":"cb36e66b3d63e7b8f94eed540320171e7d9afc85863c2c2793ff5bee2d4b1a39"} Nov 03 20:42:08.850960 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.850932 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" event={"ID":"a092ea83-8438-4626-a3a6-0e076627ab11","Type":"ContainerStarted","Data":"2e00571e81cdefa9efafff44c0ae2b4830651376f610bb24749fd6aa29e8a670"} Nov 03 20:42:08.851066 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.850979 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" event={"ID":"a092ea83-8438-4626-a3a6-0e076627ab11","Type":"ContainerStarted","Data":"6e8535d840ddaf472387e12e37ffff68011daac3337b187c04ad5d7955cabc7d"} Nov 03 20:42:08.851260 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.851246 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:08.852178 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.852159 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:08.882102 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.882080 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:08.902789 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.902764 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc"] Nov 03 20:42:08.909020 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:42:08.908996 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3fb9508_37d2_47fa_bb02_ae7735c9f408.slice/crio-a6d094f20c307d68add188d4b6745cd6e2ed6f908d6fc60c784443dff6c00f72 WatchSource:0}: Error finding container a6d094f20c307d68add188d4b6745cd6e2ed6f908d6fc60c784443dff6c00f72: Status 404 returned error can't find the container with id a6d094f20c307d68add188d4b6745cd6e2ed6f908d6fc60c784443dff6c00f72 Nov 03 20:42:08.911226 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.911197 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:42:08.911301 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.911260 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:42:08.911301 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.911288 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:42:08.911617 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.911598 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:08.959200 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:08.959148 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" podStartSLOduration=1.959133086 podStartE2EDuration="1.959133086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:08.957269319 +0000 UTC m=+800.515259837" watchObservedRunningTime="2026-11-03 20:42:08.959133086 +0000 UTC m=+800.517123585" Nov 03 20:42:09.071818 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:09.071762 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" podStartSLOduration=2.071742691 podStartE2EDuration="2.071742691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:09.069924147 +0000 UTC m=+800.627914663" watchObservedRunningTime="2026-11-03 20:42:09.071742691 +0000 UTC m=+800.629733190" Nov 03 20:42:09.303874 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:09.303828 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" podStartSLOduration=3.303812307 podStartE2EDuration="3.303812307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:09.298933564 +0000 UTC m=+800.856924084" watchObservedRunningTime="2026-11-03 20:42:09.303812307 +0000 UTC m=+800.861802805" Nov 03 20:42:09.583432 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:09.583313 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:09.589071 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:09.589048 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:09.857001 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:09.856908 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" event={"ID":"a3fb9508-37d2-47fa-bb02-ae7735c9f408","Type":"ContainerStarted","Data":"e803eaca8188de5bd61537a843c8516a7a3c35cfd45188c4e8dc7b0c19f587a1"} Nov 03 20:42:09.857001 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:09.856961 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" event={"ID":"a3fb9508-37d2-47fa-bb02-ae7735c9f408","Type":"ContainerStarted","Data":"a6d094f20c307d68add188d4b6745cd6e2ed6f908d6fc60c784443dff6c00f72"} Nov 03 20:42:09.857544 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:09.857521 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:09.858624 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:09.858602 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:42:09.886226 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:09.886209 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:09.916290 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:09.916270 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:09.967960 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:09.967922 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" podStartSLOduration=2.967910517 podStartE2EDuration="2.967910517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:09.966413574 +0000 UTC m=+801.524404091" watchObservedRunningTime="2026-11-03 20:42:09.967910517 +0000 UTC m=+801.525901015" Nov 03 20:42:10.527517 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:10.527485 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:10.532616 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:10.532596 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:10.863157 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:10.863124 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:10.864002 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:10.863189 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:10.864002 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:10.863218 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:10.864002 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:10.863477 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:10.864002 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:10.863670 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:42:10.864165 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:10.864024 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:42:18.825627 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.825573 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w"] Nov 03 20:42:18.829472 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.829454 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:18.881361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.881287 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-fe2825-openshift-default-dockercfg-7zsdg\"" Nov 03 20:42:18.896355 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.893167 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:18.896355 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.893262 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:18.896355 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.893301 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51d926d-5ce9-48da-8444-3042b674b606-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:18.896355 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.893368 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4hgn\" (UniqueName: \"kubernetes.io/projected/f51d926d-5ce9-48da-8444-3042b674b606-kube-api-access-j4hgn\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:18.896355 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.893482 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:18.896355 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.893540 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51d926d-5ce9-48da-8444-3042b674b606-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:18.896355 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.893590 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:18.896355 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.893660 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:18.896355 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.893705 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51d926d-5ce9-48da-8444-3042b674b606-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:18.928966 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.928928 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w"] Nov 03 20:42:18.994860 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.994825 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:18.994860 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.994870 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51d926d-5ce9-48da-8444-3042b674b606-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:18.995137 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.994908 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4hgn\" (UniqueName: \"kubernetes.io/projected/f51d926d-5ce9-48da-8444-3042b674b606-kube-api-access-j4hgn\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:18.995137 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.994951 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:18.995137 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.994996 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51d926d-5ce9-48da-8444-3042b674b606-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:18.995137 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.995043 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:18.995137 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.995116 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:18.995480 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.995165 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51d926d-5ce9-48da-8444-3042b674b606-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:18.995480 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.995219 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:18.996269 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.996236 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:18.996437 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.996297 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:18.996557 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.996535 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:18.996993 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.996970 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:18.997473 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.997432 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51d926d-5ce9-48da-8444-3042b674b606-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:18.999229 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:18.999205 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:19.000136 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:19.000114 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51d926d-5ce9-48da-8444-3042b674b606-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:19.049778 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:19.049749 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51d926d-5ce9-48da-8444-3042b674b606-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:19.050019 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:19.049999 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4hgn\" (UniqueName: \"kubernetes.io/projected/f51d926d-5ce9-48da-8444-3042b674b606-kube-api-access-j4hgn\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:19.144658 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:19.144569 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:19.584383 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:19.572323 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w"] Nov 03 20:42:19.616009 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:42:19.615970 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51d926d_5ce9_48da_8444_3042b674b606.slice/crio-ef9d3256b52dc5492babd7ce3a5feabac82b0ce6befab532e0ebb9f728b8613a WatchSource:0}: Error finding container ef9d3256b52dc5492babd7ce3a5feabac82b0ce6befab532e0ebb9f728b8613a: Status 404 returned error can't find the container with id ef9d3256b52dc5492babd7ce3a5feabac82b0ce6befab532e0ebb9f728b8613a Nov 03 20:42:19.618296 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:19.618263 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:42:19.618450 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:19.618359 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:42:19.618450 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:19.618409 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:42:19.919749 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:19.919669 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" event={"ID":"f51d926d-5ce9-48da-8444-3042b674b606","Type":"ContainerStarted","Data":"b258cb6acf7f1e1b407d52ef8cb5d61f055376824a5ec68e5fed5803159b2293"} Nov 03 20:42:19.919749 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:19.919734 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" event={"ID":"f51d926d-5ce9-48da-8444-3042b674b606","Type":"ContainerStarted","Data":"ef9d3256b52dc5492babd7ce3a5feabac82b0ce6befab532e0ebb9f728b8613a"} Nov 03 20:42:20.035842 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:20.035786 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" podStartSLOduration=2.035770696 podStartE2EDuration="2.035770696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:20.033219751 +0000 UTC m=+811.591210275" watchObservedRunningTime="2026-11-03 20:42:20.035770696 +0000 UTC m=+811.593761195" Nov 03 20:42:20.145508 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:20.145472 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:20.150477 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:20.150441 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:20.924537 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:20.924505 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:20.925446 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:20.925428 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:42:21.743198 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:42:21.743160 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:42:22.396609 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:22.396580 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n"] Nov 03 20:42:22.400516 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:22.400494 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n" Nov 03 20:42:22.433724 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:22.433698 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167d7ad0-634f-467c-947e-58d1f1e69fe4-maas-api-tls\") pod \"maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n\" (UID: \"167d7ad0-634f-467c-947e-58d1f1e69fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n" Nov 03 20:42:22.433844 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:22.433740 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/167d7ad0-634f-467c-947e-58d1f1e69fe4-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n\" (UID: \"167d7ad0-634f-467c-947e-58d1f1e69fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n" Nov 03 20:42:22.433901 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:22.433848 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/167d7ad0-634f-467c-947e-58d1f1e69fe4-metrics-certs\") pod \"maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n\" (UID: \"167d7ad0-634f-467c-947e-58d1f1e69fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n" Nov 03 20:42:22.433947 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:22.433927 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sj67\" (UniqueName: \"kubernetes.io/projected/167d7ad0-634f-467c-947e-58d1f1e69fe4-kube-api-access-4sj67\") pod \"maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n\" (UID: \"167d7ad0-634f-467c-947e-58d1f1e69fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n" Nov 03 20:42:22.453831 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:22.453807 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-41aa21-serving-cert\"" Nov 03 20:42:22.499865 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:22.499838 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n"] Nov 03 20:42:22.535480 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:22.535444 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sj67\" (UniqueName: \"kubernetes.io/projected/167d7ad0-634f-467c-947e-58d1f1e69fe4-kube-api-access-4sj67\") pod \"maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n\" (UID: \"167d7ad0-634f-467c-947e-58d1f1e69fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n" Nov 03 20:42:22.535703 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:22.535529 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167d7ad0-634f-467c-947e-58d1f1e69fe4-maas-api-tls\") pod \"maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n\" (UID: \"167d7ad0-634f-467c-947e-58d1f1e69fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n" Nov 03 20:42:22.535703 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:22.535554 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/167d7ad0-634f-467c-947e-58d1f1e69fe4-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n\" (UID: \"167d7ad0-634f-467c-947e-58d1f1e69fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n" Nov 03 20:42:22.535703 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:22.535598 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/167d7ad0-634f-467c-947e-58d1f1e69fe4-metrics-certs\") pod \"maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n\" (UID: \"167d7ad0-634f-467c-947e-58d1f1e69fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n" Nov 03 20:42:22.535703 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:42:22.535680 4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-serving-cert: secret "maas-api-e2e-worker-w1-41aa21-serving-cert" not found Nov 03 20:42:22.535917 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:42:22.535748 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/167d7ad0-634f-467c-947e-58d1f1e69fe4-maas-api-tls podName:167d7ad0-634f-467c-947e-58d1f1e69fe4 nodeName:}" failed. No retries permitted until 2026-11-03 20:42:23.035731924 +0000 UTC m=+814.593722402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/167d7ad0-634f-467c-947e-58d1f1e69fe4-maas-api-tls") pod "maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n" (UID: "167d7ad0-634f-467c-947e-58d1f1e69fe4") : secret "maas-api-e2e-worker-w1-41aa21-serving-cert" not found Nov 03 20:42:22.536316 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:22.536289 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/167d7ad0-634f-467c-947e-58d1f1e69fe4-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n\" (UID: \"167d7ad0-634f-467c-947e-58d1f1e69fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n" Nov 03 20:42:22.538265 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:22.538247 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/167d7ad0-634f-467c-947e-58d1f1e69fe4-metrics-certs\") pod \"maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n\" (UID: \"167d7ad0-634f-467c-947e-58d1f1e69fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n" Nov 03 20:42:22.589981 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:22.589947 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sj67\" (UniqueName: \"kubernetes.io/projected/167d7ad0-634f-467c-947e-58d1f1e69fe4-kube-api-access-4sj67\") pod \"maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n\" (UID: \"167d7ad0-634f-467c-947e-58d1f1e69fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n" Nov 03 20:42:23.041458 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:23.041421 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167d7ad0-634f-467c-947e-58d1f1e69fe4-maas-api-tls\") pod \"maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n\" (UID: \"167d7ad0-634f-467c-947e-58d1f1e69fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n" Nov 03 20:42:23.043831 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:23.043811 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167d7ad0-634f-467c-947e-58d1f1e69fe4-maas-api-tls\") pod \"maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n\" (UID: \"167d7ad0-634f-467c-947e-58d1f1e69fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n" Nov 03 20:42:23.313682 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:23.313656 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n" Nov 03 20:42:23.682310 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:23.682236 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n"] Nov 03 20:42:23.687806 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:42:23.687773 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod167d7ad0_634f_467c_947e_58d1f1e69fe4.slice/crio-bd3d80bbdddce72dcea4771adf015bf5ed1ac8dd0410e4a157edb9dac0979017 WatchSource:0}: Error finding container bd3d80bbdddce72dcea4771adf015bf5ed1ac8dd0410e4a157edb9dac0979017: Status 404 returned error can't find the container with id bd3d80bbdddce72dcea4771adf015bf5ed1ac8dd0410e4a157edb9dac0979017 Nov 03 20:42:23.945368 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:23.945252 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n" event={"ID":"167d7ad0-634f-467c-947e-58d1f1e69fe4","Type":"ContainerStarted","Data":"bd3d80bbdddce72dcea4771adf015bf5ed1ac8dd0410e4a157edb9dac0979017"} Nov 03 20:42:24.953058 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:24.952962 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n" event={"ID":"167d7ad0-634f-467c-947e-58d1f1e69fe4","Type":"ContainerStarted","Data":"a30eec0fa49e7b5e1839931a33d928a1bfce56feb205a448c1627f25a642010e"} Nov 03 20:42:24.953472 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:24.953111 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n" Nov 03 20:42:25.065838 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:25.065783 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n" podStartSLOduration=2.143061862 podStartE2EDuration="3.06576737s" totalImagesPullingTime="922.705508ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:25.064819973 +0000 UTC m=+816.622810498" watchObservedRunningTime="2026-11-03 20:42:25.06576737 +0000 UTC m=+816.623757868" Nov 03 20:42:30.964006 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:30.963981 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n" Nov 03 20:42:34.010597 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:34.010474 4493 generic.go:356] "Generic (PLEG): container finished" podID="c3be0d37-42be-4b25-846b-d8538f590441" containerID="cc23d43d708717c675b978b9771987a15b127af157120d844b8cbfb7b8120eea" exitCode=0 Nov 03 20:42:34.011255 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:34.010504 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" event={"ID":"c3be0d37-42be-4b25-846b-d8538f590441","Type":"ContainerDied","Data":"cc23d43d708717c675b978b9771987a15b127af157120d844b8cbfb7b8120eea"} Nov 03 20:42:34.011736 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:34.011718 4493 scope.go:122] "RemoveContainer" containerID="cc23d43d708717c675b978b9771987a15b127af157120d844b8cbfb7b8120eea" Nov 03 20:42:34.013316 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:34.013285 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:42:34.013450 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:34.013398 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:42:34.013515 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:34.013447 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:42:35.042616 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:35.042585 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" event={"ID":"c3be0d37-42be-4b25-846b-d8538f590441","Type":"ContainerStarted","Data":"998d8632d44a6e74b97cd33b89a524d24f21aac51ad7e6835436a00043135e2a"} Nov 03 20:42:35.881716 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:35.881666 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:35.882168 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:35.882141 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 20:42:35.882264 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:35.882207 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" podUID="c3be0d37-42be-4b25-846b-d8538f590441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 20:42:36.881552 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:36.881503 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 20:42:36.881953 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:36.881574 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" podUID="c3be0d37-42be-4b25-846b-d8538f590441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 20:42:37.881654 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:37.881617 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 20:42:37.882114 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:37.881674 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" podUID="c3be0d37-42be-4b25-846b-d8538f590441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 20:42:38.881350 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:38.881311 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 20:42:38.881532 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:38.881383 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" podUID="c3be0d37-42be-4b25-846b-d8538f590441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 20:42:39.882107 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:39.882068 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 20:42:39.882520 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:39.882128 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" podUID="c3be0d37-42be-4b25-846b-d8538f590441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 20:42:40.862849 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:40.862792 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:40.886982 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:40.886944 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:41.074355 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:41.074312 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 20:42:43.085015 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:43.084980 4493 generic.go:356] "Generic (PLEG): container finished" podID="151eca7e-97d2-4220-a630-71e7df7d9f24" containerID="6bad799127269015d6095f4181642cbb66c6026e6391f9a53413074c11664262" exitCode=0 Nov 03 20:42:43.085469 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:43.085057 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" event={"ID":"151eca7e-97d2-4220-a630-71e7df7d9f24","Type":"ContainerDied","Data":"6bad799127269015d6095f4181642cbb66c6026e6391f9a53413074c11664262"} Nov 03 20:42:43.085469 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:43.085444 4493 scope.go:122] "RemoveContainer" containerID="6bad799127269015d6095f4181642cbb66c6026e6391f9a53413074c11664262" Nov 03 20:42:43.086693 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:43.086664 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:42:43.086782 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:43.086734 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:42:43.086782 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:43.086767 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:42:44.117197 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:44.117154 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" event={"ID":"151eca7e-97d2-4220-a630-71e7df7d9f24","Type":"ContainerStarted","Data":"0041639d05478aa188687f291a834ccb726763dddf816154fc3d924d780708de"} Nov 03 20:42:44.308542 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:44.308504 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:44.309211 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:44.309180 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 20:42:44.309485 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:44.309457 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" podUID="151eca7e-97d2-4220-a630-71e7df7d9f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 20:42:45.308800 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:45.308763 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 20:42:45.309453 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:45.309420 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" podUID="151eca7e-97d2-4220-a630-71e7df7d9f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 20:42:46.313479 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:46.313442 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 20:42:46.314104 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:46.314069 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" podUID="151eca7e-97d2-4220-a630-71e7df7d9f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 20:42:47.308478 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:47.308446 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 20:42:47.308745 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:47.308718 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" podUID="151eca7e-97d2-4220-a630-71e7df7d9f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 20:42:48.306086 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:48.306056 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 20:42:48.306673 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:48.306646 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" podUID="151eca7e-97d2-4220-a630-71e7df7d9f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 20:42:49.013055 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:49.013016 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f"] Nov 03 20:42:49.016877 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:49.016856 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f" Nov 03 20:42:49.077757 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:49.077713 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f"] Nov 03 20:42:49.115150 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:49.115104 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-a19980-serving-cert\"" Nov 03 20:42:49.152301 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:49.152262 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-metrics-certs\") pod \"maas-api-e2e-worker-w6-a19980-74649996d7-54s7f\" (UID: \"fec5bf6e-548b-4ace-9533-f2f6837fa1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f" Nov 03 20:42:49.152648 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:49.152619 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfpsm\" (UniqueName: \"kubernetes.io/projected/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-kube-api-access-tfpsm\") pod \"maas-api-e2e-worker-w6-a19980-74649996d7-54s7f\" (UID: \"fec5bf6e-548b-4ace-9533-f2f6837fa1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f" Nov 03 20:42:49.152782 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:49.152672 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-maas-api-tls\") pod \"maas-api-e2e-worker-w6-a19980-74649996d7-54s7f\" (UID: \"fec5bf6e-548b-4ace-9533-f2f6837fa1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f" Nov 03 20:42:49.152845 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:49.152811 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-a19980-74649996d7-54s7f\" (UID: \"fec5bf6e-548b-4ace-9533-f2f6837fa1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f" Nov 03 20:42:49.253533 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:49.253498 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-metrics-certs\") pod \"maas-api-e2e-worker-w6-a19980-74649996d7-54s7f\" (UID: \"fec5bf6e-548b-4ace-9533-f2f6837fa1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f" Nov 03 20:42:49.253696 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:49.253555 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfpsm\" (UniqueName: \"kubernetes.io/projected/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-kube-api-access-tfpsm\") pod \"maas-api-e2e-worker-w6-a19980-74649996d7-54s7f\" (UID: \"fec5bf6e-548b-4ace-9533-f2f6837fa1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f" Nov 03 20:42:49.253696 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:49.253620 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-maas-api-tls\") pod \"maas-api-e2e-worker-w6-a19980-74649996d7-54s7f\" (UID: \"fec5bf6e-548b-4ace-9533-f2f6837fa1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f" Nov 03 20:42:49.253778 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:49.253708 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-a19980-74649996d7-54s7f\" (UID: \"fec5bf6e-548b-4ace-9533-f2f6837fa1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f" Nov 03 20:42:49.253820 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:42:49.253785 4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-serving-cert: secret "maas-api-e2e-worker-w6-a19980-serving-cert" not found Nov 03 20:42:49.253861 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:42:49.253850 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-maas-api-tls podName:fec5bf6e-548b-4ace-9533-f2f6837fa1a0 nodeName:}" failed. No retries permitted until 2026-11-03 20:42:49.75382913 +0000 UTC m=+841.311819621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-maas-api-tls") pod "maas-api-e2e-worker-w6-a19980-74649996d7-54s7f" (UID: "fec5bf6e-548b-4ace-9533-f2f6837fa1a0") : secret "maas-api-e2e-worker-w6-a19980-serving-cert" not found Nov 03 20:42:49.254650 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:49.254622 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-a19980-74649996d7-54s7f\" (UID: \"fec5bf6e-548b-4ace-9533-f2f6837fa1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f" Nov 03 20:42:49.256395 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:49.256374 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-metrics-certs\") pod \"maas-api-e2e-worker-w6-a19980-74649996d7-54s7f\" (UID: \"fec5bf6e-548b-4ace-9533-f2f6837fa1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f" Nov 03 20:42:49.308534 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:49.308505 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfpsm\" (UniqueName: \"kubernetes.io/projected/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-kube-api-access-tfpsm\") pod \"maas-api-e2e-worker-w6-a19980-74649996d7-54s7f\" (UID: \"fec5bf6e-548b-4ace-9533-f2f6837fa1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f" Nov 03 20:42:49.319849 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:49.319826 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:49.320146 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:49.320124 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:49.321026 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:49.321005 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:42:49.759633 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:49.759551 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-maas-api-tls\") pod \"maas-api-e2e-worker-w6-a19980-74649996d7-54s7f\" (UID: \"fec5bf6e-548b-4ace-9533-f2f6837fa1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f" Nov 03 20:42:49.761788 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:49.761768 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-maas-api-tls\") pod \"maas-api-e2e-worker-w6-a19980-74649996d7-54s7f\" (UID: \"fec5bf6e-548b-4ace-9533-f2f6837fa1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f" Nov 03 20:42:49.928923 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:49.928888 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f" Nov 03 20:42:50.302166 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:50.302130 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f"] Nov 03 20:42:50.306849 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:42:50.306821 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec5bf6e_548b_4ace_9533_f2f6837fa1a0.slice/crio-c651de485d1a0bf8884b2f8b542187773263839099cb837b820f33c6410f2b22 WatchSource:0}: Error finding container c651de485d1a0bf8884b2f8b542187773263839099cb837b820f33c6410f2b22: Status 404 returned error can't find the container with id c651de485d1a0bf8884b2f8b542187773263839099cb837b820f33c6410f2b22 Nov 03 20:42:51.153629 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:51.153598 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f" event={"ID":"fec5bf6e-548b-4ace-9533-f2f6837fa1a0","Type":"ContainerStarted","Data":"c651de485d1a0bf8884b2f8b542187773263839099cb837b820f33c6410f2b22"} Nov 03 20:42:51.744031 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:42:51.743983 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:42:52.160207 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:52.160171 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f" event={"ID":"fec5bf6e-548b-4ace-9533-f2f6837fa1a0","Type":"ContainerStarted","Data":"2cfd25ebf60d155b5f69a8165c5fe67348b471ec46f938c1c416ebc8dd02db82"} Nov 03 20:42:52.160627 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:52.160312 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f" Nov 03 20:42:52.269389 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:52.269319 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f" podStartSLOduration=3.396749165 podStartE2EDuration="4.269302562s" totalImagesPullingTime="872.553397ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:52.26805265 +0000 UTC m=+843.826043160" watchObservedRunningTime="2026-11-03 20:42:52.269302562 +0000 UTC m=+843.827293061" Nov 03 20:42:58.171471 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:58.171439 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f" Nov 03 20:42:59.212534 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:59.212492 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l"] Nov 03 20:42:59.216231 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:59.216204 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l" Nov 03 20:42:59.256815 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:59.256786 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-metrics-certs\") pod \"maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l\" (UID: \"fb5bc208-96eb-49b3-8501-a5f2744fd3e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l" Nov 03 20:42:59.256959 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:59.256848 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzfkz\" (UniqueName: \"kubernetes.io/projected/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-kube-api-access-vzfkz\") pod \"maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l\" (UID: \"fb5bc208-96eb-49b3-8501-a5f2744fd3e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l" Nov 03 20:42:59.256959 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:59.256940 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l\" (UID: \"fb5bc208-96eb-49b3-8501-a5f2744fd3e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l" Nov 03 20:42:59.257040 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:59.256964 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-maas-api-tls\") pod \"maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l\" (UID: \"fb5bc208-96eb-49b3-8501-a5f2744fd3e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l" Nov 03 20:42:59.263427 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:59.263399 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l"] Nov 03 20:42:59.264866 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:59.264842 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-fe2825-serving-cert\"" Nov 03 20:42:59.357854 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:59.357825 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-metrics-certs\") pod \"maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l\" (UID: \"fb5bc208-96eb-49b3-8501-a5f2744fd3e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l" Nov 03 20:42:59.358016 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:59.357900 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzfkz\" (UniqueName: \"kubernetes.io/projected/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-kube-api-access-vzfkz\") pod \"maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l\" (UID: \"fb5bc208-96eb-49b3-8501-a5f2744fd3e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l" Nov 03 20:42:59.358016 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:59.357980 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l\" (UID: \"fb5bc208-96eb-49b3-8501-a5f2744fd3e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l" Nov 03 20:42:59.358016 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:59.358007 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-maas-api-tls\") pod \"maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l\" (UID: \"fb5bc208-96eb-49b3-8501-a5f2744fd3e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l" Nov 03 20:42:59.358217 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:42:59.358199 4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-serving-cert: secret "maas-api-e2e-worker-w3-fe2825-serving-cert" not found Nov 03 20:42:59.358298 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:42:59.358282 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-maas-api-tls podName:fb5bc208-96eb-49b3-8501-a5f2744fd3e9 nodeName:}" failed. No retries permitted until 2026-11-03 20:42:59.858261598 +0000 UTC m=+851.416252103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-maas-api-tls") pod "maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l" (UID: "fb5bc208-96eb-49b3-8501-a5f2744fd3e9") : secret "maas-api-e2e-worker-w3-fe2825-serving-cert" not found Nov 03 20:42:59.359147 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:59.359122 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l\" (UID: \"fb5bc208-96eb-49b3-8501-a5f2744fd3e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l" Nov 03 20:42:59.360640 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:59.360614 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-metrics-certs\") pod \"maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l\" (UID: \"fb5bc208-96eb-49b3-8501-a5f2744fd3e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l" Nov 03 20:42:59.415081 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:59.415053 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzfkz\" (UniqueName: \"kubernetes.io/projected/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-kube-api-access-vzfkz\") pod \"maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l\" (UID: \"fb5bc208-96eb-49b3-8501-a5f2744fd3e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l" Nov 03 20:42:59.864302 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:59.864273 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-maas-api-tls\") pod \"maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l\" (UID: \"fb5bc208-96eb-49b3-8501-a5f2744fd3e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l" Nov 03 20:42:59.866748 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:42:59.866728 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-maas-api-tls\") pod \"maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l\" (UID: \"fb5bc208-96eb-49b3-8501-a5f2744fd3e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l" Nov 03 20:43:00.129623 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:00.129526 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l" Nov 03 20:43:00.509802 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:00.509712 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l"] Nov 03 20:43:00.514166 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:43:00.514136 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5bc208_96eb_49b3_8501_a5f2744fd3e9.slice/crio-1789d1e056f298fbbb6abc9c1adb22795d8d496a09637cf9a791228c81e6359e WatchSource:0}: Error finding container 1789d1e056f298fbbb6abc9c1adb22795d8d496a09637cf9a791228c81e6359e: Status 404 returned error can't find the container with id 1789d1e056f298fbbb6abc9c1adb22795d8d496a09637cf9a791228c81e6359e Nov 03 20:43:01.224182 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:01.224147 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l" event={"ID":"fb5bc208-96eb-49b3-8501-a5f2744fd3e9","Type":"ContainerStarted","Data":"1789d1e056f298fbbb6abc9c1adb22795d8d496a09637cf9a791228c81e6359e"} Nov 03 20:43:02.241126 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:02.241092 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l" event={"ID":"fb5bc208-96eb-49b3-8501-a5f2744fd3e9","Type":"ContainerStarted","Data":"4f6904e1aac1d68c7d079e907780a54e289683de4fc8cc4950b289c41f8485af"} Nov 03 20:43:02.241565 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:02.241227 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l" Nov 03 20:43:02.351442 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:02.351386 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l" podStartSLOduration=2.422666978 podStartE2EDuration="3.351371505s" totalImagesPullingTime="928.704527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:02.349322965 +0000 UTC m=+853.907313465" watchObservedRunningTime="2026-11-03 20:43:02.351371505 +0000 UTC m=+853.909362005" Nov 03 20:43:07.272482 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:07.272382 4493 generic.go:356] "Generic (PLEG): container finished" podID="a3fb9508-37d2-47fa-bb02-ae7735c9f408" containerID="e803eaca8188de5bd61537a843c8516a7a3c35cfd45188c4e8dc7b0c19f587a1" exitCode=0 Nov 03 20:43:07.272482 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:07.272426 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" event={"ID":"a3fb9508-37d2-47fa-bb02-ae7735c9f408","Type":"ContainerDied","Data":"e803eaca8188de5bd61537a843c8516a7a3c35cfd45188c4e8dc7b0c19f587a1"} Nov 03 20:43:07.272943 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:07.272796 4493 scope.go:122] "RemoveContainer" containerID="e803eaca8188de5bd61537a843c8516a7a3c35cfd45188c4e8dc7b0c19f587a1" Nov 03 20:43:07.274190 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:07.274161 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:43:07.274250 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:07.274238 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:43:07.274286 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:07.274276 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:43:08.252418 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:08.252390 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l" Nov 03 20:43:08.279529 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:08.279500 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" event={"ID":"a3fb9508-37d2-47fa-bb02-ae7735c9f408","Type":"ContainerStarted","Data":"f7006bd03df5896a554dfa898315f2fdbbdedb9a734b66264d360ccbe4488e1d"} Nov 03 20:43:08.527113 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:08.527024 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:43:08.527930 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:08.527908 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 20:43:08.528045 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:08.527953 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" podUID="a3fb9508-37d2-47fa-bb02-ae7735c9f408" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 20:43:09.528054 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:09.528015 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 20:43:09.528493 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:09.528085 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" podUID="a3fb9508-37d2-47fa-bb02-ae7735c9f408" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 20:43:10.534360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:10.529750 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 20:43:10.534360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:10.529855 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" podUID="a3fb9508-37d2-47fa-bb02-ae7735c9f408" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 20:43:10.863283 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:10.863250 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:43:11.528359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:11.527722 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 20:43:11.528359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:11.527796 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" podUID="a3fb9508-37d2-47fa-bb02-ae7735c9f408" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 20:43:12.532783 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:12.532743 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 20:43:12.533409 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:12.533371 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" podUID="a3fb9508-37d2-47fa-bb02-ae7735c9f408" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 20:43:13.581643 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:13.581602 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:43:13.583036 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:13.582967 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:43:19.385300 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:19.385263 4493 generic.go:356] "Generic (PLEG): container finished" podID="a092ea83-8438-4626-a3a6-0e076627ab11" containerID="2e00571e81cdefa9efafff44c0ae2b4830651376f610bb24749fd6aa29e8a670" exitCode=0 Nov 03 20:43:19.385861 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:19.385365 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" event={"ID":"a092ea83-8438-4626-a3a6-0e076627ab11","Type":"ContainerDied","Data":"2e00571e81cdefa9efafff44c0ae2b4830651376f610bb24749fd6aa29e8a670"} Nov 03 20:43:19.385861 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:19.385853 4493 scope.go:122] "RemoveContainer" containerID="2e00571e81cdefa9efafff44c0ae2b4830651376f610bb24749fd6aa29e8a670" Nov 03 20:43:19.387213 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:19.387181 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:43:19.387379 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:19.387264 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:43:19.387379 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:19.387315 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:43:20.417536 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:20.417497 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" event={"ID":"a092ea83-8438-4626-a3a6-0e076627ab11","Type":"ContainerStarted","Data":"9f89a58d3ee4de632bbd5db4bd9c9860bade2b236b687e01d2baa7636084ba27"} Nov 03 20:43:20.911591 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:20.911559 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:43:20.911810 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:20.911776 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 20:43:20.911888 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:20.911826 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" podUID="a092ea83-8438-4626-a3a6-0e076627ab11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 20:43:21.744014 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:43:21.743982 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:43:21.912534 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:21.912493 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 20:43:21.912728 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:21.912547 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" podUID="a092ea83-8438-4626-a3a6-0e076627ab11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 20:43:22.911946 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:22.911911 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 20:43:22.912376 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:22.911977 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" podUID="a092ea83-8438-4626-a3a6-0e076627ab11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 20:43:23.911810 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:23.911768 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 20:43:23.912145 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:23.912113 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" podUID="a092ea83-8438-4626-a3a6-0e076627ab11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 20:43:24.915357 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:24.914692 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 20:43:24.915357 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:24.914766 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" podUID="a092ea83-8438-4626-a3a6-0e076627ab11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 20:43:25.867481 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:25.867445 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:43:25.923601 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:25.923571 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:43:26.459794 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:26.459761 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:43:32.489714 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:32.489637 4493 generic.go:356] "Generic (PLEG): container finished" podID="16823f97-8616-45c4-8d13-e05b7d4bf432" containerID="c043fe533af5b8656516a7c85ade6f21fcc88348c5942b54e7aa8a3bb71674ce" exitCode=0 Nov 03 20:43:32.490141 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:32.489712 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" event={"ID":"16823f97-8616-45c4-8d13-e05b7d4bf432","Type":"ContainerDied","Data":"c043fe533af5b8656516a7c85ade6f21fcc88348c5942b54e7aa8a3bb71674ce"} Nov 03 20:43:32.490141 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:32.490096 4493 scope.go:122] "RemoveContainer" containerID="c043fe533af5b8656516a7c85ade6f21fcc88348c5942b54e7aa8a3bb71674ce" Nov 03 20:43:32.491429 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:32.491401 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:43:32.491549 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:32.491482 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:43:32.491611 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:32.491555 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:43:33.519153 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:33.519118 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" event={"ID":"16823f97-8616-45c4-8d13-e05b7d4bf432","Type":"ContainerStarted","Data":"15f7f058d7f060996cb2be070b721d0e630dc09c2c1503ed553a27d99a047045"} Nov 03 20:43:33.610360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:33.592783 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:43:33.610360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:33.592924 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 03 20:43:33.610360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:33.592969 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" podUID="16823f97-8616-45c4-8d13-e05b7d4bf432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 03 20:43:34.584302 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:34.584257 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 03 20:43:34.584950 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:34.584893 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" podUID="16823f97-8616-45c4-8d13-e05b7d4bf432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 03 20:43:35.602360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:35.596753 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 03 20:43:35.602360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:35.596834 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" podUID="16823f97-8616-45c4-8d13-e05b7d4bf432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 03 20:43:36.601476 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:36.601436 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 03 20:43:36.601765 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:36.601733 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" podUID="16823f97-8616-45c4-8d13-e05b7d4bf432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 03 20:43:37.603362 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:37.599817 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 03 20:43:37.603362 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:37.599886 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" podUID="16823f97-8616-45c4-8d13-e05b7d4bf432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 03 20:43:38.635034 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:38.635004 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:43:38.636077 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:38.635985 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:43:38.637320 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:38.637299 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 20:43:48.642457 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:48.642427 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 20:43:48.643697 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:48.643676 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 20:43:51.753659 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:43:51.753617 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:43:51.966549 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:51.966502 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:43:51.966742 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:51.966598 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" podUID="f51d926d-5ce9-48da-8444-3042b674b606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:43:52.650370 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:52.650327 4493 generic.go:356] "Generic (PLEG): container finished" podID="f51d926d-5ce9-48da-8444-3042b674b606" containerID="b258cb6acf7f1e1b407d52ef8cb5d61f055376824a5ec68e5fed5803159b2293" exitCode=0 Nov 03 20:43:52.650519 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:52.650390 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" event={"ID":"f51d926d-5ce9-48da-8444-3042b674b606","Type":"ContainerDied","Data":"b258cb6acf7f1e1b407d52ef8cb5d61f055376824a5ec68e5fed5803159b2293"} Nov 03 20:43:52.650797 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:52.650782 4493 scope.go:122] "RemoveContainer" containerID="b258cb6acf7f1e1b407d52ef8cb5d61f055376824a5ec68e5fed5803159b2293" Nov 03 20:43:52.652242 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:52.652210 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:43:52.652376 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:52.652299 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:43:52.652456 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:52.652400 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:43:53.657576 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:53.657546 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" event={"ID":"f51d926d-5ce9-48da-8444-3042b674b606","Type":"ContainerStarted","Data":"d4dadbd01850a696b7a4235c87a0c0ef4302b254c296c21a0ed4ff2ed0f2af11"} Nov 03 20:43:54.145794 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:54.145757 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:43:54.146001 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:54.145953 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 03 20:43:54.146065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:54.146003 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" podUID="f51d926d-5ce9-48da-8444-3042b674b606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 03 20:43:55.145690 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:55.145650 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 03 20:43:55.146095 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:55.145703 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" podUID="f51d926d-5ce9-48da-8444-3042b674b606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 03 20:43:56.145951 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:56.145918 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 03 20:43:56.146374 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:56.145975 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" podUID="f51d926d-5ce9-48da-8444-3042b674b606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 03 20:43:57.145003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:57.144968 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 03 20:43:57.145187 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:57.145027 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" podUID="f51d926d-5ce9-48da-8444-3042b674b606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 03 20:43:58.145060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:58.145026 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 03 20:43:58.145472 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:58.145076 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" podUID="f51d926d-5ce9-48da-8444-3042b674b606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 03 20:43:59.148966 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:59.148936 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:43:59.149493 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:59.149359 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:43:59.149979 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:43:59.149956 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:44:14.064560 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:14.064531 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m"] Nov 03 20:44:14.067460 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:14.067440 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:44:14.116814 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:14.116778 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a19980\"/\"openshift-service-ca.crt\"" Nov 03 20:44:14.163769 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:14.163733 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a19980\"/\"kube-root-ca.crt\"" Nov 03 20:44:14.163920 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:14.163800 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a19980\"/\"simulator-w6-a19980-kserve-self-signed-certs\"" Nov 03 20:44:14.163920 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:14.163797 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a19980\"/\"default-dockercfg-zj2dk\"" Nov 03 20:44:14.166219 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:14.166198 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m"] Nov 03 20:44:14.250312 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:14.250280 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-tmp-dir\") pod \"simulator-w6-a19980-kserve-586549d789-z999m\" (UID: \"46e22045-16c7-48e6-9046-87433500fe27\") " pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:44:14.250502 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:14.250362 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-home\") pod \"simulator-w6-a19980-kserve-586549d789-z999m\" (UID: \"46e22045-16c7-48e6-9046-87433500fe27\") " pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:44:14.250502 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:14.250403 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg54k\" (UniqueName: \"kubernetes.io/projected/46e22045-16c7-48e6-9046-87433500fe27-kube-api-access-mg54k\") pod \"simulator-w6-a19980-kserve-586549d789-z999m\" (UID: \"46e22045-16c7-48e6-9046-87433500fe27\") " pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:44:14.250502 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:14.250445 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-model-cache\") pod \"simulator-w6-a19980-kserve-586549d789-z999m\" (UID: \"46e22045-16c7-48e6-9046-87433500fe27\") " pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:44:14.250502 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:14.250488 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46e22045-16c7-48e6-9046-87433500fe27-tls-certs\") pod \"simulator-w6-a19980-kserve-586549d789-z999m\" (UID: \"46e22045-16c7-48e6-9046-87433500fe27\") " pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:44:14.250679 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:14.250651 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-dshm\") pod \"simulator-w6-a19980-kserve-586549d789-z999m\" (UID: \"46e22045-16c7-48e6-9046-87433500fe27\") " pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:44:14.351902 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:14.351818 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-dshm\") pod \"simulator-w6-a19980-kserve-586549d789-z999m\" (UID: \"46e22045-16c7-48e6-9046-87433500fe27\") " pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:44:14.351902 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:14.351876 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-tmp-dir\") pod \"simulator-w6-a19980-kserve-586549d789-z999m\" (UID: \"46e22045-16c7-48e6-9046-87433500fe27\") " pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:44:14.352098 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:14.351910 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-home\") pod \"simulator-w6-a19980-kserve-586549d789-z999m\" (UID: \"46e22045-16c7-48e6-9046-87433500fe27\") " pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:44:14.352098 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:14.351937 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg54k\" (UniqueName: \"kubernetes.io/projected/46e22045-16c7-48e6-9046-87433500fe27-kube-api-access-mg54k\") pod \"simulator-w6-a19980-kserve-586549d789-z999m\" (UID: \"46e22045-16c7-48e6-9046-87433500fe27\") " pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:44:14.352098 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:14.351980 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-model-cache\") pod \"simulator-w6-a19980-kserve-586549d789-z999m\" (UID: \"46e22045-16c7-48e6-9046-87433500fe27\") " pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:44:14.352098 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:14.352032 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46e22045-16c7-48e6-9046-87433500fe27-tls-certs\") pod \"simulator-w6-a19980-kserve-586549d789-z999m\" (UID: \"46e22045-16c7-48e6-9046-87433500fe27\") " pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:44:14.352461 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:14.352439 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-model-cache\") pod \"simulator-w6-a19980-kserve-586549d789-z999m\" (UID: \"46e22045-16c7-48e6-9046-87433500fe27\") " pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:44:14.352675 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:14.352593 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-home\") pod \"simulator-w6-a19980-kserve-586549d789-z999m\" (UID: \"46e22045-16c7-48e6-9046-87433500fe27\") " pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:44:14.352675 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:14.352650 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-tmp-dir\") pod \"simulator-w6-a19980-kserve-586549d789-z999m\" (UID: \"46e22045-16c7-48e6-9046-87433500fe27\") " pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:44:14.354308 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:14.354285 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-dshm\") pod \"simulator-w6-a19980-kserve-586549d789-z999m\" (UID: \"46e22045-16c7-48e6-9046-87433500fe27\") " pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:44:14.354734 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:14.354713 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46e22045-16c7-48e6-9046-87433500fe27-tls-certs\") pod \"simulator-w6-a19980-kserve-586549d789-z999m\" (UID: \"46e22045-16c7-48e6-9046-87433500fe27\") " pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:44:14.409943 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:14.409917 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg54k\" (UniqueName: \"kubernetes.io/projected/46e22045-16c7-48e6-9046-87433500fe27-kube-api-access-mg54k\") pod \"simulator-w6-a19980-kserve-586549d789-z999m\" (UID: \"46e22045-16c7-48e6-9046-87433500fe27\") " pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:44:14.678162 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:14.678074 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:44:15.047459 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:15.047427 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m"] Nov 03 20:44:15.053344 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:44:15.053298 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46e22045_16c7_48e6_9046_87433500fe27.slice/crio-fcd5aa8c1c86c1c1aca826b6c5aae5bdc2995afc4f7ccfab2a17eb7283162ce7 WatchSource:0}: Error finding container fcd5aa8c1c86c1c1aca826b6c5aae5bdc2995afc4f7ccfab2a17eb7283162ce7: Status 404 returned error can't find the container with id fcd5aa8c1c86c1c1aca826b6c5aae5bdc2995afc4f7ccfab2a17eb7283162ce7 Nov 03 20:44:15.776135 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:15.776103 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" event={"ID":"46e22045-16c7-48e6-9046-87433500fe27","Type":"ContainerStarted","Data":"4d37cb51ad870f0a92b70db7f89a6ce8a8c5d9fd32eb7dff92a982cd6add0089"} Nov 03 20:44:15.776576 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:15.776145 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" event={"ID":"46e22045-16c7-48e6-9046-87433500fe27","Type":"ContainerStarted","Data":"fcd5aa8c1c86c1c1aca826b6c5aae5bdc2995afc4f7ccfab2a17eb7283162ce7"} Nov 03 20:44:15.884848 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:15.884799 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podStartSLOduration=1.8847851370000002 podStartE2EDuration="1.884785137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:15.882600696 +0000 UTC m=+927.440591206" watchObservedRunningTime="2026-11-03 20:44:15.884785137 +0000 UTC m=+927.442775636" Nov 03 20:44:16.678720 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:16.678693 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:44:21.744454 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:44:21.744422 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:44:24.679279 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:24.679243 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:44:24.692615 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:24.692584 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:44:24.834900 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:24.834874 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:44:34.544613 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:34.544533 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv"] Nov 03 20:44:34.547459 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:34.547441 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:44:34.646162 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:34.646129 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-984cbc\"/\"openshift-service-ca.crt\"" Nov 03 20:44:34.646162 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:34.646143 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-984cbc\"/\"kube-root-ca.crt\"" Nov 03 20:44:34.646437 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:34.646187 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-984cbc\"/\"default-dockercfg-9fwld\"" Nov 03 20:44:34.646437 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:34.646227 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-984cbc\"/\"simulator-w4-984cbc-kserve-self-signed-certs\"" Nov 03 20:44:34.648629 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:34.648604 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv"] Nov 03 20:44:34.664914 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:34.664885 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-home\") pod \"simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv\" (UID: \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\") " pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:44:34.665040 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:34.664923 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmqmh\" (UniqueName: \"kubernetes.io/projected/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-kube-api-access-wmqmh\") pod \"simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv\" (UID: \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\") " pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:44:34.665040 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:34.664974 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-dshm\") pod \"simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv\" (UID: \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\") " pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:44:34.665155 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:34.665096 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-model-cache\") pod \"simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv\" (UID: \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\") " pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:44:34.665220 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:34.665203 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-tmp-dir\") pod \"simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv\" (UID: \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\") " pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:44:34.665259 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:34.665233 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-tls-certs\") pod \"simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv\" (UID: \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\") " pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:44:34.766226 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:34.766190 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-home\") pod \"simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv\" (UID: \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\") " pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:44:34.766226 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:34.766230 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmqmh\" (UniqueName: \"kubernetes.io/projected/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-kube-api-access-wmqmh\") pod \"simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv\" (UID: \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\") " pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:44:34.766510 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:34.766251 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-dshm\") pod \"simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv\" (UID: \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\") " pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:44:34.766510 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:34.766299 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-model-cache\") pod \"simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv\" (UID: \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\") " pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:44:34.766510 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:34.766376 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-tmp-dir\") pod \"simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv\" (UID: \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\") " pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:44:34.766510 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:34.766403 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-tls-certs\") pod \"simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv\" (UID: \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\") " pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:44:34.766735 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:34.766585 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-home\") pod \"simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv\" (UID: \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\") " pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:44:34.766735 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:34.766727 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-model-cache\") pod \"simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv\" (UID: \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\") " pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:44:34.766818 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:34.766767 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-tmp-dir\") pod \"simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv\" (UID: \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\") " pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:44:34.768645 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:34.768621 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-dshm\") pod \"simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv\" (UID: \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\") " pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:44:34.768944 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:34.768927 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-tls-certs\") pod \"simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv\" (UID: \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\") " pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:44:34.820366 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:34.820323 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmqmh\" (UniqueName: \"kubernetes.io/projected/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-kube-api-access-wmqmh\") pod \"simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv\" (UID: \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\") " pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:44:34.859289 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:34.859267 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:44:35.226620 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:35.226537 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv"] Nov 03 20:44:35.232346 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:44:35.232309 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6b8c002_2e0a_4f9e_86f6_23adb6f17ec1.slice/crio-573c99a6b9db4d0b22fe0e8c340118fe35f2885826b703fb7008a80716531f21 WatchSource:0}: Error finding container 573c99a6b9db4d0b22fe0e8c340118fe35f2885826b703fb7008a80716531f21: Status 404 returned error can't find the container with id 573c99a6b9db4d0b22fe0e8c340118fe35f2885826b703fb7008a80716531f21 Nov 03 20:44:35.881452 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:35.881415 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" event={"ID":"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1","Type":"ContainerStarted","Data":"174a0686fb13ccea42cff9fedd4ad081969853f31f6922ad9f961088364a7f14"} Nov 03 20:44:35.881864 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:35.881466 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" event={"ID":"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1","Type":"ContainerStarted","Data":"573c99a6b9db4d0b22fe0e8c340118fe35f2885826b703fb7008a80716531f21"} Nov 03 20:44:35.991192 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:35.991144 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podStartSLOduration=1.991130231 podStartE2EDuration="1.991130231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:35.989870687 +0000 UTC m=+947.547861198" watchObservedRunningTime="2026-11-03 20:44:35.991130231 +0000 UTC m=+947.549120730" Nov 03 20:44:36.860157 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:36.860126 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:44:44.860433 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:44.860395 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:44:44.872911 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:44.872888 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:44:44.959573 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:44:44.959544 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:44:51.744671 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:44:51.744628 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:45:00.180915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:00.180884 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z"] Nov 03 20:45:00.184620 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:00.184600 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z" Nov 03 20:45:00.233875 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:00.233852 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\"" Nov 03 20:45:00.234015 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:00.233958 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d092bd87-a80c-4e18-b600-f49ceada2355-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895645-q676z\" (UID: \"d092bd87-a80c-4e18-b600-f49ceada2355\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z" Nov 03 20:45:00.234015 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:00.234001 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzswd\" (UniqueName: \"kubernetes.io/projected/d092bd87-a80c-4e18-b600-f49ceada2355-kube-api-access-hzswd\") pod \"maas-api-key-cleanup-29895645-q676z\" (UID: \"d092bd87-a80c-4e18-b600-f49ceada2355\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z" Nov 03 20:45:00.335120 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:00.335081 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d092bd87-a80c-4e18-b600-f49ceada2355-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895645-q676z\" (UID: \"d092bd87-a80c-4e18-b600-f49ceada2355\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z" Nov 03 20:45:00.335282 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:00.335136 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzswd\" (UniqueName: \"kubernetes.io/projected/d092bd87-a80c-4e18-b600-f49ceada2355-kube-api-access-hzswd\") pod \"maas-api-key-cleanup-29895645-q676z\" (UID: \"d092bd87-a80c-4e18-b600-f49ceada2355\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z" Nov 03 20:45:00.335825 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:00.335803 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d092bd87-a80c-4e18-b600-f49ceada2355-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895645-q676z\" (UID: \"d092bd87-a80c-4e18-b600-f49ceada2355\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z" Nov 03 20:45:00.367153 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:00.367122 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z"] Nov 03 20:45:00.390015 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:00.389985 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzswd\" (UniqueName: \"kubernetes.io/projected/d092bd87-a80c-4e18-b600-f49ceada2355-kube-api-access-hzswd\") pod \"maas-api-key-cleanup-29895645-q676z\" (UID: \"d092bd87-a80c-4e18-b600-f49ceada2355\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z" Nov 03 20:45:00.497997 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:00.497917 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z" Nov 03 20:45:00.883010 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:00.882977 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z"] Nov 03 20:45:00.888383 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:45:00.888357 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd092bd87_a80c_4e18_b600_f49ceada2355.slice/crio-fb875407ec58ef709f266c4db0d324f4244f94701580effc671af4a69c0c0c7e WatchSource:0}: Error finding container fb875407ec58ef709f266c4db0d324f4244f94701580effc671af4a69c0c0c7e: Status 404 returned error can't find the container with id fb875407ec58ef709f266c4db0d324f4244f94701580effc671af4a69c0c0c7e Nov 03 20:45:01.036052 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:01.036004 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z" event={"ID":"d092bd87-a80c-4e18-b600-f49ceada2355","Type":"ContainerStarted","Data":"fb875407ec58ef709f266c4db0d324f4244f94701580effc671af4a69c0c0c7e"} Nov 03 20:45:05.063340 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:05.063303 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z" event={"ID":"d092bd87-a80c-4e18-b600-f49ceada2355","Type":"ContainerStarted","Data":"21954f3bd22c7c25ccd7eef8b66ad6c0ab5b89efb64e9288f943637f45bac582"} Nov 03 20:45:05.170458 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:05.170403 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z" podStartSLOduration=1.453112856 podStartE2EDuration="5.170383288s" totalImagesPullingTime="3.717270432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:05.168747197 +0000 UTC m=+976.726737714" watchObservedRunningTime="2026-11-03 20:45:05.170383288 +0000 UTC m=+976.728373788" Nov 03 20:45:06.070455 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:06.070419 4493 generic.go:356] "Generic (PLEG): container finished" podID="d092bd87-a80c-4e18-b600-f49ceada2355" containerID="21954f3bd22c7c25ccd7eef8b66ad6c0ab5b89efb64e9288f943637f45bac582" exitCode=60 Nov 03 20:45:06.070948 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:06.070479 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z" event={"ID":"d092bd87-a80c-4e18-b600-f49ceada2355","Type":"ContainerDied","Data":"21954f3bd22c7c25ccd7eef8b66ad6c0ab5b89efb64e9288f943637f45bac582"} Nov 03 20:45:06.070948 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:06.070809 4493 scope.go:122] "RemoveContainer" containerID="21954f3bd22c7c25ccd7eef8b66ad6c0ab5b89efb64e9288f943637f45bac582" Nov 03 20:45:07.077688 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:07.077656 4493 generic.go:356] "Generic (PLEG): container finished" podID="d092bd87-a80c-4e18-b600-f49ceada2355" containerID="f92e02faa4caa3cc58e6f9718a89ad5f6d690deb2c8d327c930714f174f0f2e9" exitCode=60 Nov 03 20:45:07.078093 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:07.077748 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z" event={"ID":"d092bd87-a80c-4e18-b600-f49ceada2355","Type":"ContainerDied","Data":"f92e02faa4caa3cc58e6f9718a89ad5f6d690deb2c8d327c930714f174f0f2e9"} Nov 03 20:45:07.078093 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:07.077805 4493 scope.go:122] "RemoveContainer" containerID="21954f3bd22c7c25ccd7eef8b66ad6c0ab5b89efb64e9288f943637f45bac582" Nov 03 20:45:07.078093 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:07.078049 4493 scope.go:122] "RemoveContainer" containerID="f92e02faa4caa3cc58e6f9718a89ad5f6d690deb2c8d327c930714f174f0f2e9" Nov 03 20:45:07.078356 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:45:07.078309 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895645-q676z_odh-ai-gateway-infra(d092bd87-a80c-4e18-b600-f49ceada2355)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z" podUID="d092bd87-a80c-4e18-b600-f49ceada2355" Nov 03 20:45:08.085104 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:08.085071 4493 scope.go:122] "RemoveContainer" containerID="f92e02faa4caa3cc58e6f9718a89ad5f6d690deb2c8d327c930714f174f0f2e9" Nov 03 20:45:08.085502 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:45:08.085317 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895645-q676z_odh-ai-gateway-infra(d092bd87-a80c-4e18-b600-f49ceada2355)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z" podUID="d092bd87-a80c-4e18-b600-f49ceada2355" Nov 03 20:45:13.439667 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:13.439630 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn"] Nov 03 20:45:13.443360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:13.443322 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:45:13.540612 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:13.540579 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b119f2\"/\"default-dockercfg-bnmdp\"" Nov 03 20:45:13.540612 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:13.540594 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b119f2\"/\"openshift-service-ca.crt\"" Nov 03 20:45:13.540838 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:13.540650 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b119f2\"/\"premium-w2-b119f2-kserve-self-signed-certs\"" Nov 03 20:45:13.540838 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:13.540706 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b119f2\"/\"kube-root-ca.crt\"" Nov 03 20:45:13.543147 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:13.543127 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn"] Nov 03 20:45:13.579497 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:13.579462 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-dshm\") pod \"premium-w2-b119f2-kserve-567bcf7f49-fthxn\" (UID: \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\") " pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:45:13.579646 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:13.579505 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dppvq\" (UniqueName: \"kubernetes.io/projected/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-kube-api-access-dppvq\") pod \"premium-w2-b119f2-kserve-567bcf7f49-fthxn\" (UID: \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\") " pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:45:13.579646 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:13.579616 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-home\") pod \"premium-w2-b119f2-kserve-567bcf7f49-fthxn\" (UID: \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\") " pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:45:13.579646 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:13.579637 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-model-cache\") pod \"premium-w2-b119f2-kserve-567bcf7f49-fthxn\" (UID: \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\") " pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:45:13.579751 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:13.579694 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-tmp-dir\") pod \"premium-w2-b119f2-kserve-567bcf7f49-fthxn\" (UID: \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\") " pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:45:13.579837 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:13.579824 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-tls-certs\") pod \"premium-w2-b119f2-kserve-567bcf7f49-fthxn\" (UID: \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\") " pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:45:13.680733 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:13.680704 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-tls-certs\") pod \"premium-w2-b119f2-kserve-567bcf7f49-fthxn\" (UID: \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\") " pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:45:13.680944 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:13.680761 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-dshm\") pod \"premium-w2-b119f2-kserve-567bcf7f49-fthxn\" (UID: \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\") " pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:45:13.680944 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:13.680805 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dppvq\" (UniqueName: \"kubernetes.io/projected/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-kube-api-access-dppvq\") pod \"premium-w2-b119f2-kserve-567bcf7f49-fthxn\" (UID: \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\") " pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:45:13.680944 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:13.680874 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-home\") pod \"premium-w2-b119f2-kserve-567bcf7f49-fthxn\" (UID: \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\") " pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:45:13.680944 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:13.680904 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-model-cache\") pod \"premium-w2-b119f2-kserve-567bcf7f49-fthxn\" (UID: \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\") " pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:45:13.681177 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:13.680976 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-tmp-dir\") pod \"premium-w2-b119f2-kserve-567bcf7f49-fthxn\" (UID: \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\") " pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:45:13.681307 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:13.681287 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-home\") pod \"premium-w2-b119f2-kserve-567bcf7f49-fthxn\" (UID: \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\") " pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:45:13.681411 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:13.681351 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-model-cache\") pod \"premium-w2-b119f2-kserve-567bcf7f49-fthxn\" (UID: \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\") " pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:45:13.681469 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:13.681410 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-tmp-dir\") pod \"premium-w2-b119f2-kserve-567bcf7f49-fthxn\" (UID: \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\") " pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:45:13.683365 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:13.683314 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-tls-certs\") pod \"premium-w2-b119f2-kserve-567bcf7f49-fthxn\" (UID: \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\") " pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:45:13.683446 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:13.683368 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-dshm\") pod \"premium-w2-b119f2-kserve-567bcf7f49-fthxn\" (UID: \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\") " pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:45:13.735003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:13.734906 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dppvq\" (UniqueName: \"kubernetes.io/projected/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-kube-api-access-dppvq\") pod \"premium-w2-b119f2-kserve-567bcf7f49-fthxn\" (UID: \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\") " pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:45:13.754250 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:13.754227 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:45:14.132133 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:14.132102 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn"] Nov 03 20:45:14.137198 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:45:14.137170 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd84d7f0a_8ffa_4211_a9be_d73ab3853b40.slice/crio-f2e8dbd23c150069b1b9f5632125a64228ed35ceaf8b33f1dcf43c65efc715ba WatchSource:0}: Error finding container f2e8dbd23c150069b1b9f5632125a64228ed35ceaf8b33f1dcf43c65efc715ba: Status 404 returned error can't find the container with id f2e8dbd23c150069b1b9f5632125a64228ed35ceaf8b33f1dcf43c65efc715ba Nov 03 20:45:15.121578 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:15.121544 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" event={"ID":"d84d7f0a-8ffa-4211-a9be-d73ab3853b40","Type":"ContainerStarted","Data":"416f6a8fded2e51851deddc103fa9b38c6b31eb80c4e14de164f4a7902da7579"} Nov 03 20:45:15.121973 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:15.121588 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" event={"ID":"d84d7f0a-8ffa-4211-a9be-d73ab3853b40","Type":"ContainerStarted","Data":"f2e8dbd23c150069b1b9f5632125a64228ed35ceaf8b33f1dcf43c65efc715ba"} Nov 03 20:45:15.230785 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:15.230735 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podStartSLOduration=2.23072146 podStartE2EDuration="2.23072146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:15.228967727 +0000 UTC m=+986.786958225" watchObservedRunningTime="2026-11-03 20:45:15.23072146 +0000 UTC m=+986.788711958" Nov 03 20:45:15.754846 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:15.754813 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:45:21.554665 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:21.554636 4493 scope.go:122] "RemoveContainer" containerID="f92e02faa4caa3cc58e6f9718a89ad5f6d690deb2c8d327c930714f174f0f2e9" Nov 03 20:45:21.807268 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:45:21.807197 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:45:22.161188 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:22.161105 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z" event={"ID":"d092bd87-a80c-4e18-b600-f49ceada2355","Type":"ContainerStarted","Data":"e627e6fb841e76323607cb34fd7663f1a92f4f872b12c43bfe1acdcccb01803a"} Nov 03 20:45:23.168803 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:23.168771 4493 generic.go:356] "Generic (PLEG): container finished" podID="d092bd87-a80c-4e18-b600-f49ceada2355" containerID="e627e6fb841e76323607cb34fd7663f1a92f4f872b12c43bfe1acdcccb01803a" exitCode=60 Nov 03 20:45:23.169315 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:23.168848 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z" event={"ID":"d092bd87-a80c-4e18-b600-f49ceada2355","Type":"ContainerDied","Data":"e627e6fb841e76323607cb34fd7663f1a92f4f872b12c43bfe1acdcccb01803a"} Nov 03 20:45:23.169315 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:23.168922 4493 scope.go:122] "RemoveContainer" containerID="f92e02faa4caa3cc58e6f9718a89ad5f6d690deb2c8d327c930714f174f0f2e9" Nov 03 20:45:23.169315 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:23.169256 4493 scope.go:122] "RemoveContainer" containerID="e627e6fb841e76323607cb34fd7663f1a92f4f872b12c43bfe1acdcccb01803a" Nov 03 20:45:23.169545 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:45:23.169527 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895645-q676z_odh-ai-gateway-infra(d092bd87-a80c-4e18-b600-f49ceada2355)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z" podUID="d092bd87-a80c-4e18-b600-f49ceada2355" Nov 03 20:45:23.278093 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:23.278062 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z"] Nov 03 20:45:23.755480 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:23.755452 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:45:23.772500 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:23.772475 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:45:24.188044 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:24.188013 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:45:24.324267 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:24.324245 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z" Nov 03 20:45:24.498464 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:24.498376 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d092bd87-a80c-4e18-b600-f49ceada2355-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d092bd87-a80c-4e18-b600-f49ceada2355-maas-api-service-ca\") pod \"d092bd87-a80c-4e18-b600-f49ceada2355\" (UID: \"d092bd87-a80c-4e18-b600-f49ceada2355\") " Nov 03 20:45:24.498464 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:24.498415 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d092bd87-a80c-4e18-b600-f49ceada2355-kube-api-access-hzswd\" (UniqueName: \"kubernetes.io/projected/d092bd87-a80c-4e18-b600-f49ceada2355-kube-api-access-hzswd\") pod \"d092bd87-a80c-4e18-b600-f49ceada2355\" (UID: \"d092bd87-a80c-4e18-b600-f49ceada2355\") " Nov 03 20:45:24.498800 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:24.498776 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d092bd87-a80c-4e18-b600-f49ceada2355-maas-api-service-ca" pod "d092bd87-a80c-4e18-b600-f49ceada2355" (UID: "d092bd87-a80c-4e18-b600-f49ceada2355"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:24.501116 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:24.501083 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d092bd87-a80c-4e18-b600-f49ceada2355-kube-api-access-hzswd" pod "d092bd87-a80c-4e18-b600-f49ceada2355" (UID: "d092bd87-a80c-4e18-b600-f49ceada2355"). InnerVolumeSpecName "kube-api-access-hzswd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:24.608269 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:24.608225 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d092bd87-a80c-4e18-b600-f49ceada2355-maas-api-service-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:24.615453 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:24.615414 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzswd\" (UniqueName: \"kubernetes.io/projected/d092bd87-a80c-4e18-b600-f49ceada2355-kube-api-access-hzswd\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:25.184548 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:25.184514 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z" Nov 03 20:45:25.186545 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:25.185031 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z" event={"ID":"d092bd87-a80c-4e18-b600-f49ceada2355","Type":"ContainerDied","Data":"fb875407ec58ef709f266c4db0d324f4244f94701580effc671af4a69c0c0c7e"} Nov 03 20:45:25.186545 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:25.185089 4493 scope.go:122] "RemoveContainer" containerID="e627e6fb841e76323607cb34fd7663f1a92f4f872b12c43bfe1acdcccb01803a" Nov 03 20:45:25.339568 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:25.339532 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z"] Nov 03 20:45:25.386235 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:25.386204 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-q676z"] Nov 03 20:45:26.561047 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:26.561015 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d092bd87-a80c-4e18-b600-f49ceada2355" path="/var/lib/kubelet/pods/d092bd87-a80c-4e18-b600-f49ceada2355/volumes" Nov 03 20:45:27.934403 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:27.934369 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6"] Nov 03 20:45:27.934885 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:27.934870 4493 cpu_manager.go:414] "containerMap: removing container" podUID="d092bd87-a80c-4e18-b600-f49ceada2355" containerName="cleanup" Nov 03 20:45:27.934943 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:27.934886 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d092bd87-a80c-4e18-b600-f49ceada2355" containerName="cleanup" Nov 03 20:45:27.934943 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:27.934899 4493 cpu_manager.go:414] "containerMap: removing container" podUID="d092bd87-a80c-4e18-b600-f49ceada2355" containerName="cleanup" Nov 03 20:45:27.934943 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:27.934904 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d092bd87-a80c-4e18-b600-f49ceada2355" containerName="cleanup" Nov 03 20:45:27.934943 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:27.934910 4493 cpu_manager.go:414] "containerMap: removing container" podUID="d092bd87-a80c-4e18-b600-f49ceada2355" containerName="cleanup" Nov 03 20:45:27.934943 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:27.934914 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d092bd87-a80c-4e18-b600-f49ceada2355" containerName="cleanup" Nov 03 20:45:27.935114 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:27.934997 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d092bd87-a80c-4e18-b600-f49ceada2355" containerName="cleanup" Nov 03 20:45:27.935114 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:27.935004 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d092bd87-a80c-4e18-b600-f49ceada2355" containerName="cleanup" Nov 03 20:45:27.938191 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:27.938172 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:45:27.951420 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:27.951400 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-tmp-dir\") pod \"premium-w6-a19980-kserve-6664698c8d-bvrq6\" (UID: \"827fd825-7e75-4174-b522-2de39a8cce28\") " pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:45:27.951521 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:27.951450 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-home\") pod \"premium-w6-a19980-kserve-6664698c8d-bvrq6\" (UID: \"827fd825-7e75-4174-b522-2de39a8cce28\") " pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:45:27.951598 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:27.951575 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-model-cache\") pod \"premium-w6-a19980-kserve-6664698c8d-bvrq6\" (UID: \"827fd825-7e75-4174-b522-2de39a8cce28\") " pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:45:27.951644 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:27.951628 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-dshm\") pod \"premium-w6-a19980-kserve-6664698c8d-bvrq6\" (UID: \"827fd825-7e75-4174-b522-2de39a8cce28\") " pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:45:27.951704 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:27.951687 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/827fd825-7e75-4174-b522-2de39a8cce28-tls-certs\") pod \"premium-w6-a19980-kserve-6664698c8d-bvrq6\" (UID: \"827fd825-7e75-4174-b522-2de39a8cce28\") " pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:45:27.951797 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:27.951783 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc2px\" (UniqueName: \"kubernetes.io/projected/827fd825-7e75-4174-b522-2de39a8cce28-kube-api-access-zc2px\") pod \"premium-w6-a19980-kserve-6664698c8d-bvrq6\" (UID: \"827fd825-7e75-4174-b522-2de39a8cce28\") " pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:45:28.035604 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:28.035573 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a19980\"/\"premium-w6-a19980-kserve-self-signed-certs\"" Nov 03 20:45:28.038305 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:28.038283 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6"] Nov 03 20:45:28.053036 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:28.053012 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-home\") pod \"premium-w6-a19980-kserve-6664698c8d-bvrq6\" (UID: \"827fd825-7e75-4174-b522-2de39a8cce28\") " pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:45:28.053169 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:28.053071 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-model-cache\") pod \"premium-w6-a19980-kserve-6664698c8d-bvrq6\" (UID: \"827fd825-7e75-4174-b522-2de39a8cce28\") " pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:45:28.053169 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:28.053092 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-dshm\") pod \"premium-w6-a19980-kserve-6664698c8d-bvrq6\" (UID: \"827fd825-7e75-4174-b522-2de39a8cce28\") " pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:45:28.053169 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:28.053107 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/827fd825-7e75-4174-b522-2de39a8cce28-tls-certs\") pod \"premium-w6-a19980-kserve-6664698c8d-bvrq6\" (UID: \"827fd825-7e75-4174-b522-2de39a8cce28\") " pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:45:28.053357 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:28.053171 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc2px\" (UniqueName: \"kubernetes.io/projected/827fd825-7e75-4174-b522-2de39a8cce28-kube-api-access-zc2px\") pod \"premium-w6-a19980-kserve-6664698c8d-bvrq6\" (UID: \"827fd825-7e75-4174-b522-2de39a8cce28\") " pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:45:28.053357 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:28.053209 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-tmp-dir\") pod \"premium-w6-a19980-kserve-6664698c8d-bvrq6\" (UID: \"827fd825-7e75-4174-b522-2de39a8cce28\") " pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:45:28.053488 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:28.053470 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-model-cache\") pod \"premium-w6-a19980-kserve-6664698c8d-bvrq6\" (UID: \"827fd825-7e75-4174-b522-2de39a8cce28\") " pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:45:28.053544 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:28.053504 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-home\") pod \"premium-w6-a19980-kserve-6664698c8d-bvrq6\" (UID: \"827fd825-7e75-4174-b522-2de39a8cce28\") " pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:45:28.053610 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:28.053595 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-tmp-dir\") pod \"premium-w6-a19980-kserve-6664698c8d-bvrq6\" (UID: \"827fd825-7e75-4174-b522-2de39a8cce28\") " pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:45:28.055430 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:28.055411 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-dshm\") pod \"premium-w6-a19980-kserve-6664698c8d-bvrq6\" (UID: \"827fd825-7e75-4174-b522-2de39a8cce28\") " pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:45:28.055697 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:28.055669 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/827fd825-7e75-4174-b522-2de39a8cce28-tls-certs\") pod \"premium-w6-a19980-kserve-6664698c8d-bvrq6\" (UID: \"827fd825-7e75-4174-b522-2de39a8cce28\") " pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:45:28.107480 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:28.107447 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc2px\" (UniqueName: \"kubernetes.io/projected/827fd825-7e75-4174-b522-2de39a8cce28-kube-api-access-zc2px\") pod \"premium-w6-a19980-kserve-6664698c8d-bvrq6\" (UID: \"827fd825-7e75-4174-b522-2de39a8cce28\") " pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:45:28.249384 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:28.249290 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:45:28.621920 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:28.621884 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6"] Nov 03 20:45:28.627713 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:45:28.627669 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod827fd825_7e75_4174_b522_2de39a8cce28.slice/crio-168bc7afee6b19459626c8c2b426e7c81a46f0bed94a31a4e34f669b7f9273cd WatchSource:0}: Error finding container 168bc7afee6b19459626c8c2b426e7c81a46f0bed94a31a4e34f669b7f9273cd: Status 404 returned error can't find the container with id 168bc7afee6b19459626c8c2b426e7c81a46f0bed94a31a4e34f669b7f9273cd Nov 03 20:45:29.210632 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:29.210594 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" event={"ID":"827fd825-7e75-4174-b522-2de39a8cce28","Type":"ContainerStarted","Data":"22ae47865f30cdb42e4034f579e425d31c4d39f0ffa80a287fa0a00cb992850b"} Nov 03 20:45:29.211070 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:29.210650 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" event={"ID":"827fd825-7e75-4174-b522-2de39a8cce28","Type":"ContainerStarted","Data":"168bc7afee6b19459626c8c2b426e7c81a46f0bed94a31a4e34f669b7f9273cd"} Nov 03 20:45:29.250242 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:29.250212 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:45:29.318467 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:29.318407 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podStartSLOduration=2.318394305 podStartE2EDuration="2.318394305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:29.317894645 +0000 UTC m=+1000.875885144" watchObservedRunningTime="2026-11-03 20:45:29.318394305 +0000 UTC m=+1000.876384804" Nov 03 20:45:38.249800 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:38.249766 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:45:38.262773 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:38.262751 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:45:38.273961 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:45:38.273933 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:45:51.744672 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:45:51.744627 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:46:20.528327 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:20.528289 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn"] Nov 03 20:46:20.528922 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:20.528564 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" podUID="16a7db53-d5c6-4339-81c4-38c917ecaa15" containerName="istio-proxy" containerID="cri-o://25d35cf4d867d9bec71cf2012596a660910a36bef4cb8a53f0f2a7b67a88012f" gracePeriod=30 Nov 03 20:46:21.743578 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:46:21.743537 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:46:25.793344 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:25.793270 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:46:25.904787 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:25.904758 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-envoy\") pod \"16a7db53-d5c6-4339-81c4-38c917ecaa15\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " Nov 03 20:46:25.904958 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:25.904820 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-podinfo\") pod \"16a7db53-d5c6-4339-81c4-38c917ecaa15\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " Nov 03 20:46:25.904958 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:25.904852 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-data\") pod \"16a7db53-d5c6-4339-81c4-38c917ecaa15\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " Nov 03 20:46:25.904958 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:25.904905 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16a7db53-d5c6-4339-81c4-38c917ecaa15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a7db53-d5c6-4339-81c4-38c917ecaa15-istiod-ca-cert\") pod \"16a7db53-d5c6-4339-81c4-38c917ecaa15\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " Nov 03 20:46:25.905115 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:25.904961 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-credential-socket\") pod \"16a7db53-d5c6-4339-81c4-38c917ecaa15\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " Nov 03 20:46:25.905115 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:25.905006 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-workload-certs\") pod \"16a7db53-d5c6-4339-81c4-38c917ecaa15\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " Nov 03 20:46:25.905115 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:25.905067 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16a7db53-d5c6-4339-81c4-38c917ecaa15-kube-api-access-5vq8f\" (UniqueName: \"kubernetes.io/projected/16a7db53-d5c6-4339-81c4-38c917ecaa15-kube-api-access-5vq8f\") pod \"16a7db53-d5c6-4339-81c4-38c917ecaa15\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " Nov 03 20:46:25.905275 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:25.905131 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-workload-socket\") pod \"16a7db53-d5c6-4339-81c4-38c917ecaa15\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " Nov 03 20:46:25.905275 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:25.905136 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-data" pod "16a7db53-d5c6-4339-81c4-38c917ecaa15" (UID: "16a7db53-d5c6-4339-81c4-38c917ecaa15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:25.905275 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:25.905180 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-token\" (UniqueName: \"kubernetes.io/projected/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-token\") pod \"16a7db53-d5c6-4339-81c4-38c917ecaa15\" (UID: \"16a7db53-d5c6-4339-81c4-38c917ecaa15\") " Nov 03 20:46:25.905275 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:25.905256 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-workload-certs" pod "16a7db53-d5c6-4339-81c4-38c917ecaa15" (UID: "16a7db53-d5c6-4339-81c4-38c917ecaa15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:25.905528 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:25.905445 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-workload-socket" pod "16a7db53-d5c6-4339-81c4-38c917ecaa15" (UID: "16a7db53-d5c6-4339-81c4-38c917ecaa15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:25.905528 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:25.905471 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16a7db53-d5c6-4339-81c4-38c917ecaa15-istiod-ca-cert" pod "16a7db53-d5c6-4339-81c4-38c917ecaa15" (UID: "16a7db53-d5c6-4339-81c4-38c917ecaa15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:46:25.905699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:25.905681 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:25.905813 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:25.905795 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a7db53-d5c6-4339-81c4-38c917ecaa15-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:25.905813 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:25.905799 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-credential-socket" pod "16a7db53-d5c6-4339-81c4-38c917ecaa15" (UID: "16a7db53-d5c6-4339-81c4-38c917ecaa15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:25.905909 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:25.905817 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:25.905909 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:25.905832 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:25.907309 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:25.907282 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-podinfo" pod "16a7db53-d5c6-4339-81c4-38c917ecaa15" (UID: "16a7db53-d5c6-4339-81c4-38c917ecaa15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:46:25.907442 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:25.907318 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-envoy" pod "16a7db53-d5c6-4339-81c4-38c917ecaa15" (UID: "16a7db53-d5c6-4339-81c4-38c917ecaa15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:25.907912 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:25.907896 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-token" pod "16a7db53-d5c6-4339-81c4-38c917ecaa15" (UID: "16a7db53-d5c6-4339-81c4-38c917ecaa15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:25.907994 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:25.907906 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16a7db53-d5c6-4339-81c4-38c917ecaa15-kube-api-access-5vq8f" pod "16a7db53-d5c6-4339-81c4-38c917ecaa15" (UID: "16a7db53-d5c6-4339-81c4-38c917ecaa15"). InnerVolumeSpecName "kube-api-access-5vq8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:26.007323 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:26.007292 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:26.007475 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:26.007322 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vq8f\" (UniqueName: \"kubernetes.io/projected/16a7db53-d5c6-4339-81c4-38c917ecaa15-kube-api-access-5vq8f\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:26.007475 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:26.007369 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:26.007475 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:26.007383 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:26.007475 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:26.007395 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a7db53-d5c6-4339-81c4-38c917ecaa15-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:26.526417 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:26.526382 4493 generic.go:356] "Generic (PLEG): container finished" podID="16a7db53-d5c6-4339-81c4-38c917ecaa15" containerID="25d35cf4d867d9bec71cf2012596a660910a36bef4cb8a53f0f2a7b67a88012f" exitCode=0 Nov 03 20:46:26.526608 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:26.526433 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" Nov 03 20:46:26.526608 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:26.526463 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" event={"ID":"16a7db53-d5c6-4339-81c4-38c917ecaa15","Type":"ContainerDied","Data":"25d35cf4d867d9bec71cf2012596a660910a36bef4cb8a53f0f2a7b67a88012f"} Nov 03 20:46:26.526608 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:26.526549 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn" event={"ID":"16a7db53-d5c6-4339-81c4-38c917ecaa15","Type":"ContainerDied","Data":"f247a19bef95e180c529f3ae38b629428002a552274a6aa4f7122459f60bec3a"} Nov 03 20:46:26.526608 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:26.526571 4493 scope.go:122] "RemoveContainer" containerID="25d35cf4d867d9bec71cf2012596a660910a36bef4cb8a53f0f2a7b67a88012f" Nov 03 20:46:26.537125 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:26.537108 4493 scope.go:122] "RemoveContainer" containerID="25d35cf4d867d9bec71cf2012596a660910a36bef4cb8a53f0f2a7b67a88012f" Nov 03 20:46:26.537392 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:46:26.537367 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25d35cf4d867d9bec71cf2012596a660910a36bef4cb8a53f0f2a7b67a88012f\": container with ID starting with 25d35cf4d867d9bec71cf2012596a660910a36bef4cb8a53f0f2a7b67a88012f not found: ID does not exist" containerID="25d35cf4d867d9bec71cf2012596a660910a36bef4cb8a53f0f2a7b67a88012f" Nov 03 20:46:26.537465 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:26.537404 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d35cf4d867d9bec71cf2012596a660910a36bef4cb8a53f0f2a7b67a88012f"} err="failed to get container status \"25d35cf4d867d9bec71cf2012596a660910a36bef4cb8a53f0f2a7b67a88012f\": rpc error: code = NotFound desc = could not find container \"25d35cf4d867d9bec71cf2012596a660910a36bef4cb8a53f0f2a7b67a88012f\": container with ID starting with 25d35cf4d867d9bec71cf2012596a660910a36bef4cb8a53f0f2a7b67a88012f not found: ID does not exist" Nov 03 20:46:26.728872 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:26.728838 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn"] Nov 03 20:46:26.732093 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:26.732067 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-bhrgn"] Nov 03 20:46:28.561529 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:46:28.561500 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16a7db53-d5c6-4339-81c4-38c917ecaa15" path="/var/lib/kubelet/pods/16a7db53-d5c6-4339-81c4-38c917ecaa15/volumes" Nov 03 20:46:51.744689 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:46:51.744654 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:47:05.879111 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:05.879073 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc"] Nov 03 20:47:05.879584 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:05.879570 4493 cpu_manager.go:414] "containerMap: removing container" podUID="16a7db53-d5c6-4339-81c4-38c917ecaa15" containerName="istio-proxy" Nov 03 20:47:05.879630 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:05.879586 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16a7db53-d5c6-4339-81c4-38c917ecaa15" containerName="istio-proxy" Nov 03 20:47:05.879671 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:05.879664 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="16a7db53-d5c6-4339-81c4-38c917ecaa15" containerName="istio-proxy" Nov 03 20:47:05.879726 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:05.879684 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d092bd87-a80c-4e18-b600-f49ceada2355" containerName="cleanup" Nov 03 20:47:05.883038 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:05.883014 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:05.925958 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:05.925928 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8"] Nov 03 20:47:05.929393 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:05.929374 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.021615 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.021579 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8"] Nov 03 20:47:06.049848 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.049816 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.050020 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.049855 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.050020 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.049974 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs98q\" (UniqueName: \"kubernetes.io/projected/ff0d6342-c880-4f83-89e5-2b5e14d512ee-kube-api-access-qs98q\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.050109 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.050029 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.050109 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.050061 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.050182 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.050120 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.050182 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.050155 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.050182 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.050179 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.050293 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.050197 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.050293 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.050242 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7hll\" (UniqueName: \"kubernetes.io/projected/c4f6d970-a5d4-4d9d-89ad-495365b7d652-kube-api-access-d7hll\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.050293 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.050290 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.050419 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.050314 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.050419 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.050359 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.050419 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.050405 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.050576 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.050453 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.050576 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.050490 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.050576 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.050509 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.050576 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.050544 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.074923 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.074874 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc"] Nov 03 20:47:06.151562 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.151464 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.151562 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.151545 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.151828 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.151578 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.151828 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.151613 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.151828 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.151679 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.151828 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.151737 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7hll\" (UniqueName: \"kubernetes.io/projected/c4f6d970-a5d4-4d9d-89ad-495365b7d652-kube-api-access-d7hll\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.151828 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.151769 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.151828 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.151806 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.152147 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.151835 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.152147 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.151870 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.152147 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.151895 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.152147 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.151934 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.152147 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.151937 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.152147 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.151956 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.152147 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.152008 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.152147 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.152017 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.152147 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.152023 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.152147 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.152100 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.152147 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.152146 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.152730 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.152246 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs98q\" (UniqueName: \"kubernetes.io/projected/ff0d6342-c880-4f83-89e5-2b5e14d512ee-kube-api-access-qs98q\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.152730 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.152275 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.152730 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.152284 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.152730 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.152287 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.152730 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.152376 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.152730 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.152549 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.152730 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.152598 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.153101 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.152881 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.153478 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.153451 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.154629 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.154605 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.154829 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.154810 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.154937 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.154921 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.155057 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.155043 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.205617 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.205586 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7hll\" (UniqueName: \"kubernetes.io/projected/c4f6d970-a5d4-4d9d-89ad-495365b7d652-kube-api-access-d7hll\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.206327 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.206306 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs98q\" (UniqueName: \"kubernetes.io/projected/ff0d6342-c880-4f83-89e5-2b5e14d512ee-kube-api-access-qs98q\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.206688 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.206662 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-5qqpc\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.206756 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.206662 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-85pk8\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.241612 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.241585 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:06.495958 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.495880 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:06.671079 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.671048 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8"] Nov 03 20:47:06.676982 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:47:06.676956 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff0d6342_c880_4f83_89e5_2b5e14d512ee.slice/crio-80a236802b9d1b2509fd76938ca142a5dbf53defaa05215c1750818d84f3613d WatchSource:0}: Error finding container 80a236802b9d1b2509fd76938ca142a5dbf53defaa05215c1750818d84f3613d: Status 404 returned error can't find the container with id 80a236802b9d1b2509fd76938ca142a5dbf53defaa05215c1750818d84f3613d Nov 03 20:47:06.679278 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.679249 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:47:06.679352 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.679316 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:47:06.679394 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.679367 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:47:06.765869 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.765836 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" event={"ID":"ff0d6342-c880-4f83-89e5-2b5e14d512ee","Type":"ContainerStarted","Data":"80a236802b9d1b2509fd76938ca142a5dbf53defaa05215c1750818d84f3613d"} Nov 03 20:47:06.869678 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.869641 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc"] Nov 03 20:47:06.874865 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:47:06.874836 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f6d970_a5d4_4d9d_89ad_495365b7d652.slice/crio-9fc902b4a9c4478138197c25fe2cd6bfb631bea6f130662256024252abfba1a6 WatchSource:0}: Error finding container 9fc902b4a9c4478138197c25fe2cd6bfb631bea6f130662256024252abfba1a6: Status 404 returned error can't find the container with id 9fc902b4a9c4478138197c25fe2cd6bfb631bea6f130662256024252abfba1a6 Nov 03 20:47:06.877371 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.877321 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:47:06.877448 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.877407 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:47:06.877448 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:06.877435 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:47:07.834357 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:07.833882 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" event={"ID":"c4f6d970-a5d4-4d9d-89ad-495365b7d652","Type":"ContainerStarted","Data":"21dae403d96d60c6909eacbc1fe3b288d73f64353f0059bd83fb18fc08193bd3"} Nov 03 20:47:07.834357 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:07.833954 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" event={"ID":"c4f6d970-a5d4-4d9d-89ad-495365b7d652","Type":"ContainerStarted","Data":"9fc902b4a9c4478138197c25fe2cd6bfb631bea6f130662256024252abfba1a6"} Nov 03 20:47:07.845357 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:07.843654 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" event={"ID":"ff0d6342-c880-4f83-89e5-2b5e14d512ee","Type":"ContainerStarted","Data":"f1dc2286ce1c2afe2160af159cad5e1cc85a5ec67d65973346ac0e9aeff3cf5c"} Nov 03 20:47:08.193626 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:08.193455 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" podStartSLOduration=3.193433674 podStartE2EDuration="3.193433674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:08.076647182 +0000 UTC m=+1099.634637696" watchObservedRunningTime="2026-11-03 20:47:08.193433674 +0000 UTC m=+1099.751424174" Nov 03 20:47:08.195313 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:08.195261 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" podStartSLOduration=3.195247777 podStartE2EDuration="3.195247777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:08.191616261 +0000 UTC m=+1099.749606787" watchObservedRunningTime="2026-11-03 20:47:08.195247777 +0000 UTC m=+1099.753238276" Nov 03 20:47:08.244706 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:08.244674 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:08.277535 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:08.276453 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-85pk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 20:47:08.277535 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:08.276516 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" podUID="ff0d6342-c880-4f83-89e5-2b5e14d512ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 20:47:08.506422 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:08.506152 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:08.506721 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:08.506693 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-5qqpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 20:47:08.506898 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:08.506871 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" podUID="c4f6d970-a5d4-4d9d-89ad-495365b7d652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 20:47:09.242072 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:09.242029 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-85pk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 20:47:09.242523 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:09.242356 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" podUID="ff0d6342-c880-4f83-89e5-2b5e14d512ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 20:47:09.504410 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:09.503737 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-5qqpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 20:47:09.504410 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:09.503800 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" podUID="c4f6d970-a5d4-4d9d-89ad-495365b7d652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 20:47:10.241879 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:10.241853 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-85pk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 20:47:10.242115 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:10.242096 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" podUID="ff0d6342-c880-4f83-89e5-2b5e14d512ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 20:47:10.496841 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:10.496746 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-5qqpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 20:47:10.497101 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:10.497070 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" podUID="c4f6d970-a5d4-4d9d-89ad-495365b7d652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 20:47:11.243480 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:11.242674 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-85pk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 20:47:11.243480 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:11.242748 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" podUID="ff0d6342-c880-4f83-89e5-2b5e14d512ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 20:47:11.506417 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:11.503695 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-5qqpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 20:47:11.506417 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:11.503760 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" podUID="c4f6d970-a5d4-4d9d-89ad-495365b7d652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 20:47:12.242998 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:12.242953 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-85pk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 20:47:12.243183 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:12.243018 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" podUID="ff0d6342-c880-4f83-89e5-2b5e14d512ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 20:47:12.498001 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:12.497897 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-5qqpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 20:47:12.498641 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:12.498591 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" podUID="c4f6d970-a5d4-4d9d-89ad-495365b7d652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 20:47:13.242838 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:13.242805 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-85pk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 20:47:13.242986 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:13.242857 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" podUID="ff0d6342-c880-4f83-89e5-2b5e14d512ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 20:47:13.496500 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:13.496406 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-5qqpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 20:47:13.496500 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:13.496472 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" podUID="c4f6d970-a5d4-4d9d-89ad-495365b7d652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 20:47:14.242353 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:14.242301 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-85pk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 20:47:14.242877 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:14.242390 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" podUID="ff0d6342-c880-4f83-89e5-2b5e14d512ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 20:47:14.497001 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:14.496907 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-5qqpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 20:47:14.497001 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:14.496986 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" podUID="c4f6d970-a5d4-4d9d-89ad-495365b7d652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 20:47:15.458780 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:15.458745 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:15.459319 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:15.459222 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:15.460238 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:15.460212 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:47:15.695776 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:15.695732 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:15.696713 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:15.696506 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:15.700124 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:15.700100 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:47:21.743737 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:47:21.743690 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:47:24.538300 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.538264 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r"] Nov 03 20:47:24.542394 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.542370 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.603503 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.603469 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.603695 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.603517 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.603695 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.603603 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.603695 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.603661 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.603893 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.603746 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.603893 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.603762 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.603893 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.603778 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7w6r\" (UniqueName: \"kubernetes.io/projected/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-kube-api-access-z7w6r\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.603893 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.603811 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.603893 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.603826 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.643291 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.643260 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r"] Nov 03 20:47:24.704556 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.704527 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.704729 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.704563 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.704729 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.704585 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.704729 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.704655 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.704892 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.704740 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.704892 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.704765 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.704892 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.704789 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7w6r\" (UniqueName: \"kubernetes.io/projected/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-kube-api-access-z7w6r\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.704892 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.704846 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.704892 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.704873 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.705139 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.705012 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.705440 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.705417 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.705591 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.705484 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.705672 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.705505 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.705672 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.705610 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.707706 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.707668 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.707795 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.707710 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.758103 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.758074 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.758387 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.758367 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7w6r\" (UniqueName: \"kubernetes.io/projected/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-kube-api-access-z7w6r\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:24.860547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:24.860510 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:25.102037 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.101994 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv"] Nov 03 20:47:25.107042 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.107021 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.163963 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.163936 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-41aa21-openshift-default-dockercfg-plbb2\"" Nov 03 20:47:25.210626 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.210589 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.210810 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.210687 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.210810 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.210787 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.210932 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.210823 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.210932 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.210891 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2b5f\" (UniqueName: \"kubernetes.io/projected/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-kube-api-access-h2b5f\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.211034 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.210957 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.211034 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.210987 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.211140 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.211081 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.211194 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.211156 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.212947 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.212924 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv"] Nov 03 20:47:25.289117 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.289086 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r"] Nov 03 20:47:25.294177 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:47:25.294146 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8dec249_9f2a_4bfd_b83b_f72d50e5b936.slice/crio-60809dd2298ffdb7153f93d64832e55c3dab201a92a7a5a5bef985831c5dc005 WatchSource:0}: Error finding container 60809dd2298ffdb7153f93d64832e55c3dab201a92a7a5a5bef985831c5dc005: Status 404 returned error can't find the container with id 60809dd2298ffdb7153f93d64832e55c3dab201a92a7a5a5bef985831c5dc005 Nov 03 20:47:25.296940 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.296910 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:47:25.297029 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.296982 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:47:25.297029 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.297010 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:47:25.311826 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.311805 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.311935 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.311853 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.311935 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.311874 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.311935 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.311916 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.312172 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.311950 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.312172 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.312004 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2b5f\" (UniqueName: \"kubernetes.io/projected/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-kube-api-access-h2b5f\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.312172 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.312064 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.312172 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.312107 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.312172 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.312150 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.312453 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.312428 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.312620 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.312601 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.312966 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.312940 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.313035 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.312966 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.313127 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.313099 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.314525 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.314502 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.315218 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.315182 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.366282 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.366234 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.366408 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.366283 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2b5f\" (UniqueName: \"kubernetes.io/projected/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-kube-api-access-h2b5f\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.422415 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.422044 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:25.864938 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.864900 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv"] Nov 03 20:47:25.870273 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:47:25.870247 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a14d5e_7956_4ffe_9a82_3ad0de7cb4ba.slice/crio-0181cd59e438fa0ed681cce5bd061ca664f6fe760a80f40cbe6d9aa735271f26 WatchSource:0}: Error finding container 0181cd59e438fa0ed681cce5bd061ca664f6fe760a80f40cbe6d9aa735271f26: Status 404 returned error can't find the container with id 0181cd59e438fa0ed681cce5bd061ca664f6fe760a80f40cbe6d9aa735271f26 Nov 03 20:47:25.872846 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.872805 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:47:25.872955 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.872877 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:47:25.872955 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.872905 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:47:25.958222 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.958186 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" event={"ID":"f8dec249-9f2a-4bfd-b83b-f72d50e5b936","Type":"ContainerStarted","Data":"742bad4705c31054cc620bed93507bf6120cc4755ecc17274c4da1d3e5a2e4de"} Nov 03 20:47:25.958405 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.958253 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" event={"ID":"f8dec249-9f2a-4bfd-b83b-f72d50e5b936","Type":"ContainerStarted","Data":"60809dd2298ffdb7153f93d64832e55c3dab201a92a7a5a5bef985831c5dc005"} Nov 03 20:47:25.965714 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.963588 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" event={"ID":"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba","Type":"ContainerStarted","Data":"7c69dfcb738986768ace5780daba2d9bbe8f77170fbff8d768cc7acffbe59263"} Nov 03 20:47:25.965714 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:25.963717 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" event={"ID":"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba","Type":"ContainerStarted","Data":"0181cd59e438fa0ed681cce5bd061ca664f6fe760a80f40cbe6d9aa735271f26"} Nov 03 20:47:26.069989 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:26.069931 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" podStartSLOduration=2.069916123 podStartE2EDuration="2.069916123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:26.067238464 +0000 UTC m=+1117.625228980" watchObservedRunningTime="2026-11-03 20:47:26.069916123 +0000 UTC m=+1117.627906622" Nov 03 20:47:26.187323 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:26.187204 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" podStartSLOduration=1.187185178 podStartE2EDuration="1.187185178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:26.180857716 +0000 UTC m=+1117.738848255" watchObservedRunningTime="2026-11-03 20:47:26.187185178 +0000 UTC m=+1117.745175678" Nov 03 20:47:26.422600 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:26.422558 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:26.424154 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:26.424128 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 03 20:47:26.424282 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:26.424177 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" podUID="14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 03 20:47:26.861572 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:26.861546 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:26.863228 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:26.863203 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 20:47:26.863370 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:26.863247 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" podUID="f8dec249-9f2a-4bfd-b83b-f72d50e5b936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 20:47:27.435603 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:27.435090 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 03 20:47:27.435603 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:27.435150 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" podUID="14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 03 20:47:27.861741 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:27.861706 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 20:47:27.861921 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:27.861762 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" podUID="f8dec249-9f2a-4bfd-b83b-f72d50e5b936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 20:47:28.424747 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:28.424709 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 03 20:47:28.425045 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:28.425019 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" podUID="14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 03 20:47:28.860906 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:28.860864 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 20:47:28.861425 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:28.860919 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" podUID="f8dec249-9f2a-4bfd-b83b-f72d50e5b936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 20:47:29.425359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:29.424691 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 03 20:47:29.425359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:29.424753 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" podUID="14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 03 20:47:29.861086 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:29.861054 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 20:47:29.861511 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:29.861108 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" podUID="f8dec249-9f2a-4bfd-b83b-f72d50e5b936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 20:47:30.431759 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:30.431724 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 03 20:47:30.432052 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:30.432022 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" podUID="14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 03 20:47:30.861783 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:30.861745 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 20:47:30.862265 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:30.861807 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" podUID="f8dec249-9f2a-4bfd-b83b-f72d50e5b936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 20:47:31.424359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:31.424000 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 03 20:47:31.424359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:31.424089 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" podUID="14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 03 20:47:31.867357 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:31.863786 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 20:47:31.867357 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:31.863849 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" podUID="f8dec249-9f2a-4bfd-b83b-f72d50e5b936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 20:47:32.467949 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:32.458993 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 03 20:47:32.467949 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:32.459070 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" podUID="14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 03 20:47:33.153502 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:33.153459 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:33.154042 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:33.153906 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:33.154756 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:33.154737 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 20:47:34.288031 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:34.288001 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:34.288562 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:34.288535 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:34.290364 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:47:34.289315 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 20:47:51.757722 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:47:51.757684 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:48:10.271534 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.271498 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d"] Nov 03 20:48:10.276263 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.276242 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:48:10.304344 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.304314 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-dshm\") pod \"distinct-w5-1ec892-kserve-6496b67d7d-27f2d\" (UID: \"590f00b6-8a20-45ff-8a02-b7e456071057\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:48:10.304451 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.304410 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590f00b6-8a20-45ff-8a02-b7e456071057-tls-certs\") pod \"distinct-w5-1ec892-kserve-6496b67d7d-27f2d\" (UID: \"590f00b6-8a20-45ff-8a02-b7e456071057\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:48:10.304497 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.304473 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-home\") pod \"distinct-w5-1ec892-kserve-6496b67d7d-27f2d\" (UID: \"590f00b6-8a20-45ff-8a02-b7e456071057\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:48:10.304532 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.304516 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-model-cache\") pod \"distinct-w5-1ec892-kserve-6496b67d7d-27f2d\" (UID: \"590f00b6-8a20-45ff-8a02-b7e456071057\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:48:10.304646 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.304631 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-tmp-dir\") pod \"distinct-w5-1ec892-kserve-6496b67d7d-27f2d\" (UID: \"590f00b6-8a20-45ff-8a02-b7e456071057\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:48:10.304741 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.304726 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpd5w\" (UniqueName: \"kubernetes.io/projected/590f00b6-8a20-45ff-8a02-b7e456071057-kube-api-access-rpd5w\") pod \"distinct-w5-1ec892-kserve-6496b67d7d-27f2d\" (UID: \"590f00b6-8a20-45ff-8a02-b7e456071057\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:48:10.329825 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.329792 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d"] Nov 03 20:48:10.372062 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.372035 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-1ec892\"/\"openshift-service-ca.crt\"" Nov 03 20:48:10.372288 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.372267 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-1ec892\"/\"default-dockercfg-sx87j\"" Nov 03 20:48:10.372388 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.372302 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-1ec892\"/\"kube-root-ca.crt\"" Nov 03 20:48:10.372439 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.372410 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-1ec892\"/\"distinct-w5-1ec892-kserve-self-signed-certs\"" Nov 03 20:48:10.405324 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.405300 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpd5w\" (UniqueName: \"kubernetes.io/projected/590f00b6-8a20-45ff-8a02-b7e456071057-kube-api-access-rpd5w\") pod \"distinct-w5-1ec892-kserve-6496b67d7d-27f2d\" (UID: \"590f00b6-8a20-45ff-8a02-b7e456071057\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:48:10.405483 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.405361 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-dshm\") pod \"distinct-w5-1ec892-kserve-6496b67d7d-27f2d\" (UID: \"590f00b6-8a20-45ff-8a02-b7e456071057\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:48:10.405483 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.405386 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590f00b6-8a20-45ff-8a02-b7e456071057-tls-certs\") pod \"distinct-w5-1ec892-kserve-6496b67d7d-27f2d\" (UID: \"590f00b6-8a20-45ff-8a02-b7e456071057\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:48:10.405483 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.405425 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-home\") pod \"distinct-w5-1ec892-kserve-6496b67d7d-27f2d\" (UID: \"590f00b6-8a20-45ff-8a02-b7e456071057\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:48:10.405483 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.405476 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-model-cache\") pod \"distinct-w5-1ec892-kserve-6496b67d7d-27f2d\" (UID: \"590f00b6-8a20-45ff-8a02-b7e456071057\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:48:10.405695 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.405550 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-tmp-dir\") pod \"distinct-w5-1ec892-kserve-6496b67d7d-27f2d\" (UID: \"590f00b6-8a20-45ff-8a02-b7e456071057\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:48:10.405916 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.405893 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-home\") pod \"distinct-w5-1ec892-kserve-6496b67d7d-27f2d\" (UID: \"590f00b6-8a20-45ff-8a02-b7e456071057\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:48:10.405916 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.405904 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-model-cache\") pod \"distinct-w5-1ec892-kserve-6496b67d7d-27f2d\" (UID: \"590f00b6-8a20-45ff-8a02-b7e456071057\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:48:10.406098 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.405961 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-tmp-dir\") pod \"distinct-w5-1ec892-kserve-6496b67d7d-27f2d\" (UID: \"590f00b6-8a20-45ff-8a02-b7e456071057\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:48:10.407736 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.407717 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-dshm\") pod \"distinct-w5-1ec892-kserve-6496b67d7d-27f2d\" (UID: \"590f00b6-8a20-45ff-8a02-b7e456071057\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:48:10.408043 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.408027 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590f00b6-8a20-45ff-8a02-b7e456071057-tls-certs\") pod \"distinct-w5-1ec892-kserve-6496b67d7d-27f2d\" (UID: \"590f00b6-8a20-45ff-8a02-b7e456071057\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:48:10.472987 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.472953 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpd5w\" (UniqueName: \"kubernetes.io/projected/590f00b6-8a20-45ff-8a02-b7e456071057-kube-api-access-rpd5w\") pod \"distinct-w5-1ec892-kserve-6496b67d7d-27f2d\" (UID: \"590f00b6-8a20-45ff-8a02-b7e456071057\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:48:10.590838 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.590809 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:48:10.968723 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:10.968640 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d"] Nov 03 20:48:10.973600 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:48:10.973574 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod590f00b6_8a20_45ff_8a02_b7e456071057.slice/crio-ff5ec9dcc2c781a1a8b66bbaeef4a3298e1c5c8f9233cfb30a03146464ff282d WatchSource:0}: Error finding container ff5ec9dcc2c781a1a8b66bbaeef4a3298e1c5c8f9233cfb30a03146464ff282d: Status 404 returned error can't find the container with id ff5ec9dcc2c781a1a8b66bbaeef4a3298e1c5c8f9233cfb30a03146464ff282d Nov 03 20:48:11.541563 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:11.541530 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" event={"ID":"590f00b6-8a20-45ff-8a02-b7e456071057","Type":"ContainerStarted","Data":"c7564cc61b0556ffe55481e9d2d5b45e68edfd2a509b20d80876ccf8a16885fb"} Nov 03 20:48:11.541563 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:11.541569 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" event={"ID":"590f00b6-8a20-45ff-8a02-b7e456071057","Type":"ContainerStarted","Data":"ff5ec9dcc2c781a1a8b66bbaeef4a3298e1c5c8f9233cfb30a03146464ff282d"} Nov 03 20:48:11.591031 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:11.590999 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:48:11.648998 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:11.648931 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podStartSLOduration=1.648915787 podStartE2EDuration="1.648915787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:11.648633913 +0000 UTC m=+1163.206624437" watchObservedRunningTime="2026-11-03 20:48:11.648915787 +0000 UTC m=+1163.206906287" Nov 03 20:48:20.591550 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:20.591300 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:48:20.609051 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:20.609015 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:48:20.621072 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:20.621044 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:48:21.744567 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:48:21.744534 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:48:23.127892 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.127853 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn"] Nov 03 20:48:23.131974 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.131953 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:48:23.228556 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.228525 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-fe2825\"/\"openshift-service-ca.crt\"" Nov 03 20:48:23.228556 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.228558 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-fe2825\"/\"kube-root-ca.crt\"" Nov 03 20:48:23.228781 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.228634 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-fe2825\"/\"default-dockercfg-sv9gn\"" Nov 03 20:48:23.228823 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.228788 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-fe2825\"/\"distinct-w3-fe2825-kserve-self-signed-certs\"" Nov 03 20:48:23.231352 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.231323 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn"] Nov 03 20:48:23.257681 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.257658 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s74t\" (UniqueName: \"kubernetes.io/projected/70624c1d-a6b0-4131-97dc-1774fda506ad-kube-api-access-5s74t\") pod \"distinct-w3-fe2825-kserve-59d4747868-cldgn\" (UID: \"70624c1d-a6b0-4131-97dc-1774fda506ad\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:48:23.257805 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.257687 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-dshm\") pod \"distinct-w3-fe2825-kserve-59d4747868-cldgn\" (UID: \"70624c1d-a6b0-4131-97dc-1774fda506ad\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:48:23.257805 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.257713 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-model-cache\") pod \"distinct-w3-fe2825-kserve-59d4747868-cldgn\" (UID: \"70624c1d-a6b0-4131-97dc-1774fda506ad\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:48:23.257805 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.257776 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-tmp-dir\") pod \"distinct-w3-fe2825-kserve-59d4747868-cldgn\" (UID: \"70624c1d-a6b0-4131-97dc-1774fda506ad\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:48:23.257912 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.257847 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70624c1d-a6b0-4131-97dc-1774fda506ad-tls-certs\") pod \"distinct-w3-fe2825-kserve-59d4747868-cldgn\" (UID: \"70624c1d-a6b0-4131-97dc-1774fda506ad\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:48:23.257946 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.257928 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-home\") pod \"distinct-w3-fe2825-kserve-59d4747868-cldgn\" (UID: \"70624c1d-a6b0-4131-97dc-1774fda506ad\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:48:23.358884 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.358842 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-home\") pod \"distinct-w3-fe2825-kserve-59d4747868-cldgn\" (UID: \"70624c1d-a6b0-4131-97dc-1774fda506ad\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:48:23.359061 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.359005 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s74t\" (UniqueName: \"kubernetes.io/projected/70624c1d-a6b0-4131-97dc-1774fda506ad-kube-api-access-5s74t\") pod \"distinct-w3-fe2825-kserve-59d4747868-cldgn\" (UID: \"70624c1d-a6b0-4131-97dc-1774fda506ad\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:48:23.359061 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.359037 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-dshm\") pod \"distinct-w3-fe2825-kserve-59d4747868-cldgn\" (UID: \"70624c1d-a6b0-4131-97dc-1774fda506ad\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:48:23.359240 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.359215 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-model-cache\") pod \"distinct-w3-fe2825-kserve-59d4747868-cldgn\" (UID: \"70624c1d-a6b0-4131-97dc-1774fda506ad\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:48:23.359288 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.359258 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-tmp-dir\") pod \"distinct-w3-fe2825-kserve-59d4747868-cldgn\" (UID: \"70624c1d-a6b0-4131-97dc-1774fda506ad\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:48:23.359288 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.359270 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-home\") pod \"distinct-w3-fe2825-kserve-59d4747868-cldgn\" (UID: \"70624c1d-a6b0-4131-97dc-1774fda506ad\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:48:23.359386 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.359354 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70624c1d-a6b0-4131-97dc-1774fda506ad-tls-certs\") pod \"distinct-w3-fe2825-kserve-59d4747868-cldgn\" (UID: \"70624c1d-a6b0-4131-97dc-1774fda506ad\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:48:23.359590 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.359569 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-tmp-dir\") pod \"distinct-w3-fe2825-kserve-59d4747868-cldgn\" (UID: \"70624c1d-a6b0-4131-97dc-1774fda506ad\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:48:23.359699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.359632 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-model-cache\") pod \"distinct-w3-fe2825-kserve-59d4747868-cldgn\" (UID: \"70624c1d-a6b0-4131-97dc-1774fda506ad\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:48:23.361411 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.361391 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-dshm\") pod \"distinct-w3-fe2825-kserve-59d4747868-cldgn\" (UID: \"70624c1d-a6b0-4131-97dc-1774fda506ad\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:48:23.361850 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.361833 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70624c1d-a6b0-4131-97dc-1774fda506ad-tls-certs\") pod \"distinct-w3-fe2825-kserve-59d4747868-cldgn\" (UID: \"70624c1d-a6b0-4131-97dc-1774fda506ad\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:48:23.413391 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.413313 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s74t\" (UniqueName: \"kubernetes.io/projected/70624c1d-a6b0-4131-97dc-1774fda506ad-kube-api-access-5s74t\") pod \"distinct-w3-fe2825-kserve-59d4747868-cldgn\" (UID: \"70624c1d-a6b0-4131-97dc-1774fda506ad\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:48:23.444035 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.444013 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:48:23.827998 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:23.827957 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn"] Nov 03 20:48:23.832279 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:48:23.832254 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70624c1d_a6b0_4131_97dc_1774fda506ad.slice/crio-2f0ba972cce19c348f1182c16156f124f1c35220b890a7efd8df6a17b91d38dc WatchSource:0}: Error finding container 2f0ba972cce19c348f1182c16156f124f1c35220b890a7efd8df6a17b91d38dc: Status 404 returned error can't find the container with id 2f0ba972cce19c348f1182c16156f124f1c35220b890a7efd8df6a17b91d38dc Nov 03 20:48:24.629258 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:24.629224 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" event={"ID":"70624c1d-a6b0-4131-97dc-1774fda506ad","Type":"ContainerStarted","Data":"83b7b2649e6d9bdb960537f02bf9f5a6a071cb5bfb9e12b7c8cc5cc621b4deda"} Nov 03 20:48:24.629727 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:24.629267 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" event={"ID":"70624c1d-a6b0-4131-97dc-1774fda506ad","Type":"ContainerStarted","Data":"2f0ba972cce19c348f1182c16156f124f1c35220b890a7efd8df6a17b91d38dc"} Nov 03 20:48:24.739164 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:24.739105 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podStartSLOduration=1.7390911359999999 podStartE2EDuration="1.739091136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:24.73587682 +0000 UTC m=+1176.293867344" watchObservedRunningTime="2026-11-03 20:48:24.739091136 +0000 UTC m=+1176.297081686" Nov 03 20:48:25.444485 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:25.444456 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:48:33.445074 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:33.445026 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:48:33.458377 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:33.458353 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:48:33.702047 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:33.701959 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:48:36.753881 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:36.753847 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67"] Nov 03 20:48:36.757751 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:36.757733 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:48:36.835937 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:36.835906 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86tll\" (UniqueName: \"kubernetes.io/projected/50477b11-be5c-4d14-b7ad-663dd674f82e-kube-api-access-86tll\") pod \"distinct-w0-fd9afe-kserve-56449c6948-hzd67\" (UID: \"50477b11-be5c-4d14-b7ad-663dd674f82e\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:48:36.836132 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:36.835967 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-tmp-dir\") pod \"distinct-w0-fd9afe-kserve-56449c6948-hzd67\" (UID: \"50477b11-be5c-4d14-b7ad-663dd674f82e\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:48:36.836132 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:36.836100 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-home\") pod \"distinct-w0-fd9afe-kserve-56449c6948-hzd67\" (UID: \"50477b11-be5c-4d14-b7ad-663dd674f82e\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:48:36.836242 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:36.836201 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50477b11-be5c-4d14-b7ad-663dd674f82e-tls-certs\") pod \"distinct-w0-fd9afe-kserve-56449c6948-hzd67\" (UID: \"50477b11-be5c-4d14-b7ad-663dd674f82e\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:48:36.836294 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:36.836278 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-model-cache\") pod \"distinct-w0-fd9afe-kserve-56449c6948-hzd67\" (UID: \"50477b11-be5c-4d14-b7ad-663dd674f82e\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:48:36.836366 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:36.836298 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-dshm\") pod \"distinct-w0-fd9afe-kserve-56449c6948-hzd67\" (UID: \"50477b11-be5c-4d14-b7ad-663dd674f82e\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:48:36.838804 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:36.838779 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67"] Nov 03 20:48:36.854947 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:36.854924 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fd9afe\"/\"distinct-w0-fd9afe-kserve-self-signed-certs\"" Nov 03 20:48:36.854947 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:36.854925 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fd9afe\"/\"default-dockercfg-596g2\"" Nov 03 20:48:36.876805 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:36.876783 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-fd9afe\"/\"openshift-service-ca.crt\"" Nov 03 20:48:36.876918 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:36.876819 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-fd9afe\"/\"kube-root-ca.crt\"" Nov 03 20:48:36.938383 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:36.938344 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-home\") pod \"distinct-w0-fd9afe-kserve-56449c6948-hzd67\" (UID: \"50477b11-be5c-4d14-b7ad-663dd674f82e\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:48:36.938598 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:36.938446 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50477b11-be5c-4d14-b7ad-663dd674f82e-tls-certs\") pod \"distinct-w0-fd9afe-kserve-56449c6948-hzd67\" (UID: \"50477b11-be5c-4d14-b7ad-663dd674f82e\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:48:36.938598 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:36.938561 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-model-cache\") pod \"distinct-w0-fd9afe-kserve-56449c6948-hzd67\" (UID: \"50477b11-be5c-4d14-b7ad-663dd674f82e\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:48:36.938715 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:36.938590 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-dshm\") pod \"distinct-w0-fd9afe-kserve-56449c6948-hzd67\" (UID: \"50477b11-be5c-4d14-b7ad-663dd674f82e\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:48:36.938767 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:36.938747 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86tll\" (UniqueName: \"kubernetes.io/projected/50477b11-be5c-4d14-b7ad-663dd674f82e-kube-api-access-86tll\") pod \"distinct-w0-fd9afe-kserve-56449c6948-hzd67\" (UID: \"50477b11-be5c-4d14-b7ad-663dd674f82e\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:48:36.939108 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:36.938827 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-tmp-dir\") pod \"distinct-w0-fd9afe-kserve-56449c6948-hzd67\" (UID: \"50477b11-be5c-4d14-b7ad-663dd674f82e\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:48:36.939108 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:36.938853 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-home\") pod \"distinct-w0-fd9afe-kserve-56449c6948-hzd67\" (UID: \"50477b11-be5c-4d14-b7ad-663dd674f82e\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:48:36.939108 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:36.938951 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-model-cache\") pod \"distinct-w0-fd9afe-kserve-56449c6948-hzd67\" (UID: \"50477b11-be5c-4d14-b7ad-663dd674f82e\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:48:36.939273 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:36.939121 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-tmp-dir\") pod \"distinct-w0-fd9afe-kserve-56449c6948-hzd67\" (UID: \"50477b11-be5c-4d14-b7ad-663dd674f82e\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:48:36.941671 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:36.941644 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50477b11-be5c-4d14-b7ad-663dd674f82e-tls-certs\") pod \"distinct-w0-fd9afe-kserve-56449c6948-hzd67\" (UID: \"50477b11-be5c-4d14-b7ad-663dd674f82e\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:48:36.942288 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:36.942270 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-dshm\") pod \"distinct-w0-fd9afe-kserve-56449c6948-hzd67\" (UID: \"50477b11-be5c-4d14-b7ad-663dd674f82e\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:48:36.993812 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:36.993783 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86tll\" (UniqueName: \"kubernetes.io/projected/50477b11-be5c-4d14-b7ad-663dd674f82e-kube-api-access-86tll\") pod \"distinct-w0-fd9afe-kserve-56449c6948-hzd67\" (UID: \"50477b11-be5c-4d14-b7ad-663dd674f82e\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:48:37.070240 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:37.070207 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:48:37.464628 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:37.464548 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67"] Nov 03 20:48:37.469597 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:48:37.469566 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50477b11_be5c_4d14_b7ad_663dd674f82e.slice/crio-977f64257f908f57611b5eb14d29d021e946c5c93e811e42ad230240386cbd87 WatchSource:0}: Error finding container 977f64257f908f57611b5eb14d29d021e946c5c93e811e42ad230240386cbd87: Status 404 returned error can't find the container with id 977f64257f908f57611b5eb14d29d021e946c5c93e811e42ad230240386cbd87 Nov 03 20:48:37.708297 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:37.708265 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" event={"ID":"50477b11-be5c-4d14-b7ad-663dd674f82e","Type":"ContainerStarted","Data":"767ac478a1db0e744c9a4ad9746eb9e05c880ed5e6f8c0ee595e724fa695e312"} Nov 03 20:48:37.708297 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:37.708300 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" event={"ID":"50477b11-be5c-4d14-b7ad-663dd674f82e","Type":"ContainerStarted","Data":"977f64257f908f57611b5eb14d29d021e946c5c93e811e42ad230240386cbd87"} Nov 03 20:48:37.822014 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:37.821957 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podStartSLOduration=1.8219436500000001 podStartE2EDuration="1.82194365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:37.820816641 +0000 UTC m=+1189.378807160" watchObservedRunningTime="2026-11-03 20:48:37.82194365 +0000 UTC m=+1189.379934148" Nov 03 20:48:38.071142 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:38.071110 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:48:47.070970 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:47.070937 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:48:47.083155 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:47.083132 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:48:47.785566 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:47.785538 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:48:48.691161 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:48.691120 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 20:48:48.692851 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:48:48.692825 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 20:48:51.743427 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:48:51.743385 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:49:17.323951 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:17.323916 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn"] Nov 03 20:49:17.327743 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:17.327728 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:49:17.424537 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:17.424512 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a19980\"/\"distinct-2-w6-a19980-kserve-self-signed-certs\"" Nov 03 20:49:17.434606 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:17.434569 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-dshm\") pod \"distinct-2-w6-a19980-kserve-68ccb9958d-9njbn\" (UID: \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:49:17.434962 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:17.434938 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dnxw\" (UniqueName: \"kubernetes.io/projected/e8205aa0-9b8a-4961-bd95-947ac5edfdff-kube-api-access-6dnxw\") pod \"distinct-2-w6-a19980-kserve-68ccb9958d-9njbn\" (UID: \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:49:17.435061 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:17.435029 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8205aa0-9b8a-4961-bd95-947ac5edfdff-tls-certs\") pod \"distinct-2-w6-a19980-kserve-68ccb9958d-9njbn\" (UID: \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:49:17.435126 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:17.435078 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-model-cache\") pod \"distinct-2-w6-a19980-kserve-68ccb9958d-9njbn\" (UID: \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:49:17.435185 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:17.435131 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-tmp-dir\") pod \"distinct-2-w6-a19980-kserve-68ccb9958d-9njbn\" (UID: \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:49:17.435294 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:17.435278 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-home\") pod \"distinct-2-w6-a19980-kserve-68ccb9958d-9njbn\" (UID: \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:49:17.437530 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:17.437507 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn"] Nov 03 20:49:17.536547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:17.536508 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dnxw\" (UniqueName: \"kubernetes.io/projected/e8205aa0-9b8a-4961-bd95-947ac5edfdff-kube-api-access-6dnxw\") pod \"distinct-2-w6-a19980-kserve-68ccb9958d-9njbn\" (UID: \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:49:17.536763 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:17.536571 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8205aa0-9b8a-4961-bd95-947ac5edfdff-tls-certs\") pod \"distinct-2-w6-a19980-kserve-68ccb9958d-9njbn\" (UID: \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:49:17.536763 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:17.536599 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-model-cache\") pod \"distinct-2-w6-a19980-kserve-68ccb9958d-9njbn\" (UID: \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:49:17.536763 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:17.536628 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-tmp-dir\") pod \"distinct-2-w6-a19980-kserve-68ccb9958d-9njbn\" (UID: \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:49:17.536763 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:17.536650 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-home\") pod \"distinct-2-w6-a19980-kserve-68ccb9958d-9njbn\" (UID: \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:49:17.536763 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:17.536707 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-dshm\") pod \"distinct-2-w6-a19980-kserve-68ccb9958d-9njbn\" (UID: \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:49:17.537113 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:17.537086 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-model-cache\") pod \"distinct-2-w6-a19980-kserve-68ccb9958d-9njbn\" (UID: \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:49:17.537113 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:17.537104 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-tmp-dir\") pod \"distinct-2-w6-a19980-kserve-68ccb9958d-9njbn\" (UID: \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:49:17.537323 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:17.537165 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-home\") pod \"distinct-2-w6-a19980-kserve-68ccb9958d-9njbn\" (UID: \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:49:17.539216 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:17.539192 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-dshm\") pod \"distinct-2-w6-a19980-kserve-68ccb9958d-9njbn\" (UID: \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:49:17.539389 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:17.539373 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8205aa0-9b8a-4961-bd95-947ac5edfdff-tls-certs\") pod \"distinct-2-w6-a19980-kserve-68ccb9958d-9njbn\" (UID: \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:49:17.590600 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:17.590540 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dnxw\" (UniqueName: \"kubernetes.io/projected/e8205aa0-9b8a-4961-bd95-947ac5edfdff-kube-api-access-6dnxw\") pod \"distinct-2-w6-a19980-kserve-68ccb9958d-9njbn\" (UID: \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:49:17.639699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:17.639669 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:49:18.031604 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:18.031508 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn"] Nov 03 20:49:18.037093 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:49:18.037067 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8205aa0_9b8a_4961_bd95_947ac5edfdff.slice/crio-804f18dcfa8402a5decfa6d0bf01ffbf3d8edd51bdb220ba44f18b0b2d67d816 WatchSource:0}: Error finding container 804f18dcfa8402a5decfa6d0bf01ffbf3d8edd51bdb220ba44f18b0b2d67d816: Status 404 returned error can't find the container with id 804f18dcfa8402a5decfa6d0bf01ffbf3d8edd51bdb220ba44f18b0b2d67d816 Nov 03 20:49:18.958166 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:18.958124 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" event={"ID":"e8205aa0-9b8a-4961-bd95-947ac5edfdff","Type":"ContainerStarted","Data":"9a67360724ee710140e42327eb5c36a330ea3c6898dde54aebbe5f9f70ab2618"} Nov 03 20:49:18.958659 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:18.958176 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" event={"ID":"e8205aa0-9b8a-4961-bd95-947ac5edfdff","Type":"ContainerStarted","Data":"804f18dcfa8402a5decfa6d0bf01ffbf3d8edd51bdb220ba44f18b0b2d67d816"} Nov 03 20:49:19.068972 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:19.068922 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podStartSLOduration=2.068907843 podStartE2EDuration="2.068907843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:49:19.066829164 +0000 UTC m=+1230.624819664" watchObservedRunningTime="2026-11-03 20:49:19.068907843 +0000 UTC m=+1230.626898361" Nov 03 20:49:19.640431 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:19.640399 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:49:21.745464 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:49:21.745428 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:49:27.640687 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:27.640651 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:49:27.653377 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:27.653351 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:49:28.062216 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:28.062192 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:49:39.567237 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.567207 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b"] Nov 03 20:49:39.571309 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.571291 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.670942 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.670915 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b"] Nov 03 20:49:39.680695 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.680667 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.680835 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.680740 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.680899 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.680874 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.680956 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.680926 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnmh6\" (UniqueName: \"kubernetes.io/projected/57aeddbf-a41f-4cc0-8f76-d589898c2144-kube-api-access-rnmh6\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.681129 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.681107 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.681167 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.681142 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.681203 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.681166 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57aeddbf-a41f-4cc0-8f76-d589898c2144-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.681236 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.681221 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.681271 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.681250 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.782633 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.782602 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.782799 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.782662 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.782799 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.782694 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnmh6\" (UniqueName: \"kubernetes.io/projected/57aeddbf-a41f-4cc0-8f76-d589898c2144-kube-api-access-rnmh6\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.782799 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.782773 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.782960 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.782799 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.782960 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.782830 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57aeddbf-a41f-4cc0-8f76-d589898c2144-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.782960 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.782866 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.782960 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.782906 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.783168 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.782978 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.783226 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.783199 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.783295 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.783243 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.783711 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.783648 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.783855 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.783829 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57aeddbf-a41f-4cc0-8f76-d589898c2144-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.783984 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.783958 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.785202 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.785177 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.785408 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.785386 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.839094 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.839003 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnmh6\" (UniqueName: \"kubernetes.io/projected/57aeddbf-a41f-4cc0-8f76-d589898c2144-kube-api-access-rnmh6\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.842955 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.842933 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:39.885786 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:39.885753 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:40.317420 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:40.317388 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b"] Nov 03 20:49:40.322161 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:49:40.322139 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57aeddbf_a41f_4cc0_8f76_d589898c2144.slice/crio-b7c42e2b74a7f5722673a70fc0b732c51656ea1e475291793e9f6633a7a83670 WatchSource:0}: Error finding container b7c42e2b74a7f5722673a70fc0b732c51656ea1e475291793e9f6633a7a83670: Status 404 returned error can't find the container with id b7c42e2b74a7f5722673a70fc0b732c51656ea1e475291793e9f6633a7a83670 Nov 03 20:49:40.324572 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:40.324543 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:49:40.324644 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:40.324617 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:49:40.324681 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:40.324647 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:49:41.118818 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:41.118783 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" event={"ID":"57aeddbf-a41f-4cc0-8f76-d589898c2144","Type":"ContainerStarted","Data":"9571d5e8003ba1be713011a8d27c1a7b39ddd944bbc126a134c01dff54c10474"} Nov 03 20:49:41.119248 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:41.118844 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" event={"ID":"57aeddbf-a41f-4cc0-8f76-d589898c2144","Type":"ContainerStarted","Data":"b7c42e2b74a7f5722673a70fc0b732c51656ea1e475291793e9f6633a7a83670"} Nov 03 20:49:41.323477 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:41.323422 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" podStartSLOduration=2.323406718 podStartE2EDuration="2.323406718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:49:41.274827682 +0000 UTC m=+1252.832818201" watchObservedRunningTime="2026-11-03 20:49:41.323406718 +0000 UTC m=+1252.881397293" Nov 03 20:49:41.902357 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:41.899449 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 20:49:41.902357 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:41.899516 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" podUID="57aeddbf-a41f-4cc0-8f76-d589898c2144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 20:49:41.902357 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:41.899812 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:42.905781 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:42.905741 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 20:49:42.906407 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:42.906377 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" podUID="57aeddbf-a41f-4cc0-8f76-d589898c2144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 20:49:43.902358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:43.893731 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 20:49:43.902358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:43.893807 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" podUID="57aeddbf-a41f-4cc0-8f76-d589898c2144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 20:49:44.887706 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:44.887671 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 20:49:44.888308 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:44.888273 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" podUID="57aeddbf-a41f-4cc0-8f76-d589898c2144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 20:49:45.890723 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:45.890684 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 20:49:45.894361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:45.891294 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" podUID="57aeddbf-a41f-4cc0-8f76-d589898c2144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 20:49:46.886903 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:46.886864 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 20:49:46.887085 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:46.886919 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" podUID="57aeddbf-a41f-4cc0-8f76-d589898c2144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 20:49:47.890711 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:47.890674 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:47.891119 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:47.891100 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:47.891682 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:47.891656 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:49:51.744906 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:49:51.744870 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:49:56.182638 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.182606 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779"] Nov 03 20:49:56.186955 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.186933 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.287566 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.287534 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779"] Nov 03 20:49:56.292898 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.292868 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.293065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.293007 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.293065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.293034 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.293186 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.293151 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.293263 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.293242 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.293323 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.293297 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtrzr\" (UniqueName: \"kubernetes.io/projected/156d066a-c5e2-469e-86b3-b92e319f71a0-kube-api-access-qtrzr\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.293460 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.293442 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.293526 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.293489 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/156d066a-c5e2-469e-86b3-b92e319f71a0-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.293590 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.293571 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.394273 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.394239 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.394273 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.394272 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.394519 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.394392 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.394519 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.394450 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.394519 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.394496 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtrzr\" (UniqueName: \"kubernetes.io/projected/156d066a-c5e2-469e-86b3-b92e319f71a0-kube-api-access-qtrzr\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.394669 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.394569 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.394669 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.394589 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/156d066a-c5e2-469e-86b3-b92e319f71a0-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.394669 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.394644 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.394970 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.394936 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.395274 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.394636 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.395274 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.394999 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.395274 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.395091 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.395496 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.395313 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.395496 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.395393 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/156d066a-c5e2-469e-86b3-b92e319f71a0-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.396931 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.396910 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.397030 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.396969 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.447924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.447865 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.448031 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.447989 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtrzr\" (UniqueName: \"kubernetes.io/projected/156d066a-c5e2-469e-86b3-b92e319f71a0-kube-api-access-qtrzr\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.499537 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.499511 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:56.929859 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.929829 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779"] Nov 03 20:49:56.936200 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.936162 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:49:56.936415 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.936254 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:49:56.936415 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:56.936305 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:49:57.228617 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:57.228532 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" event={"ID":"156d066a-c5e2-469e-86b3-b92e319f71a0","Type":"ContainerStarted","Data":"eb8984806b24485991a8d76b1f72a5941419d1029abf0dec676d8404daac6508"} Nov 03 20:49:57.228617 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:57.228594 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" event={"ID":"156d066a-c5e2-469e-86b3-b92e319f71a0","Type":"ContainerStarted","Data":"c216ecb47206096641e4b55b82d05b98e1969eb92383325f24dda6dcf34b3aee"} Nov 03 20:49:57.340877 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:57.340824 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" podStartSLOduration=1.340808699 podStartE2EDuration="1.340808699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:49:57.336818047 +0000 UTC m=+1268.894808570" watchObservedRunningTime="2026-11-03 20:49:57.340808699 +0000 UTC m=+1268.898799263" Nov 03 20:49:57.507462 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:57.503414 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:49:58.509679 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:58.509643 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:49:58.510187 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:58.509698 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" podUID="156d066a-c5e2-469e-86b3-b92e319f71a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 20:49:59.503358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:59.502698 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:49:59.503358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:49:59.502759 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" podUID="156d066a-c5e2-469e-86b3-b92e319f71a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 20:50:00.507357 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:00.501712 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:50:00.507357 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:00.501773 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" podUID="156d066a-c5e2-469e-86b3-b92e319f71a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 20:50:01.500756 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:01.500719 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:50:01.501055 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:01.501026 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" podUID="156d066a-c5e2-469e-86b3-b92e319f71a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 20:50:02.508525 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:02.508484 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:50:02.509017 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:02.508558 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" podUID="156d066a-c5e2-469e-86b3-b92e319f71a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 20:50:03.500228 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:03.500182 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:50:03.500458 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:03.500238 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" podUID="156d066a-c5e2-469e-86b3-b92e319f71a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 20:50:04.504574 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:04.504542 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:50:04.505015 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:04.504838 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:50:04.505485 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:04.505469 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:50:21.745235 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:50:21.745201 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:50:24.577797 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.577764 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g"] Nov 03 20:50:24.582039 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.582018 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.659072 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.658078 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.659072 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.658131 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.659072 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.658169 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.659072 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.658388 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.659072 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.658467 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.659072 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.659073 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.659551 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.659115 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.660643 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.660620 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cbx8\" (UniqueName: \"kubernetes.io/projected/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-kube-api-access-2cbx8\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.660770 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.660701 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.687086 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.687057 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g"] Nov 03 20:50:24.762394 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.762346 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.762394 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.762403 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cbx8\" (UniqueName: \"kubernetes.io/projected/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-kube-api-access-2cbx8\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.762667 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.762455 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.762667 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.762507 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.762667 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.762543 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.762667 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.762567 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.763225 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.762671 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.763225 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.762694 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.763225 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.762764 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.763225 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.762912 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.763225 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.762930 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.763614 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.763243 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.763614 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.763318 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.763614 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.763404 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.765169 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.765142 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.765718 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.765690 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.816409 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.816382 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.816563 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.816546 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cbx8\" (UniqueName: \"kubernetes.io/projected/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-kube-api-access-2cbx8\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:24.898226 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:24.898150 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:25.323930 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:25.323863 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g"] Nov 03 20:50:25.370003 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:50:25.369976 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd923b0_6c5e_40d8_b5ba_d3c8e87d24eb.slice/crio-6ba62f3a0b3ccab39ac47a92a1e86a4e1b665797a1598a81f441163c92344849 WatchSource:0}: Error finding container 6ba62f3a0b3ccab39ac47a92a1e86a4e1b665797a1598a81f441163c92344849: Status 404 returned error can't find the container with id 6ba62f3a0b3ccab39ac47a92a1e86a4e1b665797a1598a81f441163c92344849 Nov 03 20:50:25.372267 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:25.372235 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:50:25.372382 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:25.372321 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:50:25.372458 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:25.372399 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:50:25.405798 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:25.405757 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" event={"ID":"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb","Type":"ContainerStarted","Data":"6ba62f3a0b3ccab39ac47a92a1e86a4e1b665797a1598a81f441163c92344849"} Nov 03 20:50:26.415471 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:26.415430 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" event={"ID":"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb","Type":"ContainerStarted","Data":"b43663a8986a28c18666a2fdc3d0b8219cc91a0d79e30927f017f3e81b59d44f"} Nov 03 20:50:26.574293 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:26.574243 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" podStartSLOduration=2.574229056 podStartE2EDuration="2.574229056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:26.571195478 +0000 UTC m=+1298.129185995" watchObservedRunningTime="2026-11-03 20:50:26.574229056 +0000 UTC m=+1298.132219554" Nov 03 20:50:26.908733 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:26.905667 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:26.909609 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:26.909574 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:50:26.909744 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:26.909642 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" podUID="fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 20:50:27.904358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:27.902741 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:50:27.904358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:27.902815 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" podUID="fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 20:50:28.902420 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:28.898731 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:50:28.902420 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:28.898792 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" podUID="fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 20:50:29.907358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:29.906696 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:50:29.907358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:29.906754 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" podUID="fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 20:50:30.908494 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:30.908441 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:50:30.909002 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:30.908513 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" podUID="fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 20:50:31.899278 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:31.899246 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:50:31.899463 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:31.899319 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" podUID="fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 20:50:32.902641 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:32.902614 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:32.903053 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:32.902976 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:32.903567 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:32.903550 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:50:41.540549 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.540498 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm"] Nov 03 20:50:41.576290 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.576260 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.703852 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.703814 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm"] Nov 03 20:50:41.720417 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.718630 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.720417 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.718687 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.720417 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.718747 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.720417 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.718773 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c79419-fc27-4ffb-8222-63e30302cef4-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.720417 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.718828 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzxgm\" (UniqueName: \"kubernetes.io/projected/48c79419-fc27-4ffb-8222-63e30302cef4-kube-api-access-kzxgm\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.720417 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.718890 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c79419-fc27-4ffb-8222-63e30302cef4-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.720417 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.719012 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c79419-fc27-4ffb-8222-63e30302cef4-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.720417 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.719051 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.720417 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.719089 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.820287 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.820242 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c79419-fc27-4ffb-8222-63e30302cef4-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.820287 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.820293 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.820581 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.820439 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.820581 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.820525 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.820581 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.820561 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.820762 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.820615 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.820762 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.820637 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c79419-fc27-4ffb-8222-63e30302cef4-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.820762 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.820691 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzxgm\" (UniqueName: \"kubernetes.io/projected/48c79419-fc27-4ffb-8222-63e30302cef4-kube-api-access-kzxgm\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.820762 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.820759 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c79419-fc27-4ffb-8222-63e30302cef4-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.821018 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.820990 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c79419-fc27-4ffb-8222-63e30302cef4-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.821325 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.821292 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.821475 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.821415 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.821676 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.821651 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.821822 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.821653 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.823076 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.823049 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.824115 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.824091 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c79419-fc27-4ffb-8222-63e30302cef4-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.874743 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.874716 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c79419-fc27-4ffb-8222-63e30302cef4-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.875634 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.875613 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzxgm\" (UniqueName: \"kubernetes.io/projected/48c79419-fc27-4ffb-8222-63e30302cef4-kube-api-access-kzxgm\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:41.964519 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:41.964486 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:42.450889 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:42.450859 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm"] Nov 03 20:50:42.452218 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:42.452185 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:50:42.452306 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:42.452249 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:50:42.452306 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:42.452278 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:50:42.586027 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:42.585980 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" event={"ID":"48c79419-fc27-4ffb-8222-63e30302cef4","Type":"ContainerStarted","Data":"0f03cd8d3f2fc52c722f700adc7a918ffdbde06257c42bb1f473f71fc5fbe371"} Nov 03 20:50:42.586660 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:42.586635 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" event={"ID":"48c79419-fc27-4ffb-8222-63e30302cef4","Type":"ContainerStarted","Data":"b751a7580e9575d84f2504e77170ebc397ed6cb43410746be56916f0416550db"} Nov 03 20:50:42.698727 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:42.698630 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" podStartSLOduration=1.698616671 podStartE2EDuration="1.698616671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:42.696783504 +0000 UTC m=+1314.254774021" watchObservedRunningTime="2026-11-03 20:50:42.698616671 +0000 UTC m=+1314.256607170" Nov 03 20:50:42.965372 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:42.965268 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:43.967431 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:43.967402 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 20:50:43.967814 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:43.967453 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" podUID="48c79419-fc27-4ffb-8222-63e30302cef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 20:50:44.965304 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:44.965258 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 20:50:44.965538 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:44.965327 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" podUID="48c79419-fc27-4ffb-8222-63e30302cef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 20:50:45.965829 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:45.965789 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 20:50:45.966207 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:45.965852 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" podUID="48c79419-fc27-4ffb-8222-63e30302cef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 20:50:46.965781 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:46.965731 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 20:50:46.965995 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:46.965817 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" podUID="48c79419-fc27-4ffb-8222-63e30302cef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 20:50:47.964805 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:47.964769 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 20:50:47.964984 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:47.964824 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" podUID="48c79419-fc27-4ffb-8222-63e30302cef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 20:50:48.965363 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:48.965297 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 20:50:48.965781 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:48.965385 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" podUID="48c79419-fc27-4ffb-8222-63e30302cef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 20:50:49.969161 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:49.969132 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:49.969585 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:49.969434 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:49.970023 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:50:49.970005 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:50:51.744131 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:50:51.744093 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:51:00.535468 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:00.535431 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm"] Nov 03 20:51:00.540541 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:00.540523 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:51:00.634803 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:00.634769 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm"] Nov 03 20:51:00.637167 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:00.637145 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-984cbc\"/\"distinct-2-w4-984cbc-kserve-self-signed-certs\"" Nov 03 20:51:00.705495 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:00.705465 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-dshm\") pod \"distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm\" (UID: \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:51:00.705658 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:00.705524 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-home\") pod \"distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm\" (UID: \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:51:00.705658 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:00.705620 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-model-cache\") pod \"distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm\" (UID: \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:51:00.705787 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:00.705769 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-tmp-dir\") pod \"distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm\" (UID: \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:51:00.705896 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:00.705882 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-tls-certs\") pod \"distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm\" (UID: \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:51:00.705951 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:00.705903 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9k62\" (UniqueName: \"kubernetes.io/projected/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-kube-api-access-b9k62\") pod \"distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm\" (UID: \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:51:00.807218 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:00.807188 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-dshm\") pod \"distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm\" (UID: \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:51:00.807441 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:00.807274 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-home\") pod \"distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm\" (UID: \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:51:00.807441 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:00.807326 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-model-cache\") pod \"distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm\" (UID: \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:51:00.807581 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:00.807466 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-tmp-dir\") pod \"distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm\" (UID: \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:51:00.807636 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:00.807579 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-tls-certs\") pod \"distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm\" (UID: \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:51:00.807636 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:00.807606 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9k62\" (UniqueName: \"kubernetes.io/projected/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-kube-api-access-b9k62\") pod \"distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm\" (UID: \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:51:00.807855 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:00.807825 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-home\") pod \"distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm\" (UID: \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:51:00.808008 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:00.807960 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-tmp-dir\") pod \"distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm\" (UID: \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:51:00.808093 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:00.808075 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-model-cache\") pod \"distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm\" (UID: \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:51:00.809803 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:00.809782 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-dshm\") pod \"distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm\" (UID: \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:51:00.810237 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:00.810218 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-tls-certs\") pod \"distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm\" (UID: \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:51:00.862479 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:00.862451 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9k62\" (UniqueName: \"kubernetes.io/projected/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-kube-api-access-b9k62\") pod \"distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm\" (UID: \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:51:01.152593 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:01.152505 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:51:01.530856 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:01.530764 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm"] Nov 03 20:51:01.536347 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:51:01.536305 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e9a83e5_1798_4ac7_9675_99d0caa9cde6.slice/crio-c1e588aa14558e8fc57d97e1f75b5dfb9bf9da0293998a73aaf04dde4d4207ba WatchSource:0}: Error finding container c1e588aa14558e8fc57d97e1f75b5dfb9bf9da0293998a73aaf04dde4d4207ba: Status 404 returned error can't find the container with id c1e588aa14558e8fc57d97e1f75b5dfb9bf9da0293998a73aaf04dde4d4207ba Nov 03 20:51:01.729131 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:01.729091 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" event={"ID":"2e9a83e5-1798-4ac7-9675-99d0caa9cde6","Type":"ContainerStarted","Data":"48bab4f0f58ac15df0d057dbf3c8d46688cb7c8a79ea2b1f67141164e4f53b2e"} Nov 03 20:51:01.729418 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:01.729142 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" event={"ID":"2e9a83e5-1798-4ac7-9675-99d0caa9cde6","Type":"ContainerStarted","Data":"c1e588aa14558e8fc57d97e1f75b5dfb9bf9da0293998a73aaf04dde4d4207ba"} Nov 03 20:51:01.838054 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:01.838009 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podStartSLOduration=1.837995531 podStartE2EDuration="1.837995531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:01.835847581 +0000 UTC m=+1333.393838080" watchObservedRunningTime="2026-11-03 20:51:01.837995531 +0000 UTC m=+1333.395986071" Nov 03 20:51:02.152901 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:02.152815 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:51:09.671360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:09.670451 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:09.910359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:09.906495 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 20:51:09.910359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:09.906887 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 03 20:51:09.957416 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:09.956811 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:10.058463 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:10.058419 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:10.183401 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:10.179413 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:10.292929 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:10.292840 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:11.153465 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:11.153429 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:51:11.166222 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:11.166194 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:51:11.888456 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:11.888427 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:51:21.743921 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:51:21.743878 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:51:26.842827 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:26.842794 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf"] Nov 03 20:51:26.847450 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:26.847426 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:26.889643 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:26.889619 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx"] Nov 03 20:51:26.893440 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:26.893419 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:26.989414 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:26.989384 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx"] Nov 03 20:51:27.016504 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.016480 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.016504 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.016512 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vf27\" (UniqueName: \"kubernetes.io/projected/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-kube-api-access-8vf27\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.016707 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.016541 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.016750 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.016687 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.016798 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.016754 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.016798 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.016783 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.016881 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.016810 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldhtf\" (UniqueName: \"kubernetes.io/projected/038c8d67-7cd3-48bf-a359-4ff2c577b32c-kube-api-access-ldhtf\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.016881 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.016861 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.017029 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.017011 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.017124 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.017109 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.017170 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.017160 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.017267 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.017249 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.017308 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.017283 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.017382 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.017311 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.017382 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.017350 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.017471 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.017412 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.017518 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.017480 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.017518 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.017504 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.040125 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.040100 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf"] Nov 03 20:51:27.118723 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.118633 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.118723 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.118674 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.118947 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.118740 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.118947 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.118764 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.118947 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.118900 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.118947 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.118933 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.119168 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.118986 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.119168 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.119033 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.119168 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.119069 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.119168 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.119098 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.119168 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.119124 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.119168 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.119152 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.119168 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.119156 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vf27\" (UniqueName: \"kubernetes.io/projected/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-kube-api-access-8vf27\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.119619 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.119204 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.119619 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.119252 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.119619 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.119286 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.119619 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.119323 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.119619 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.119367 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.119619 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.119392 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldhtf\" (UniqueName: \"kubernetes.io/projected/038c8d67-7cd3-48bf-a359-4ff2c577b32c-kube-api-access-ldhtf\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.119619 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.119430 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.119619 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.119528 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.119619 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.119541 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.119619 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.119608 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.120091 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.119640 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.120091 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.119249 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.120091 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.119911 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.120091 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.119959 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.120250 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.120113 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.122030 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.122004 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.122133 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.122072 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.122458 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.122432 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.122637 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.122619 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.172413 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.172387 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.173257 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.173232 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldhtf\" (UniqueName: \"kubernetes.io/projected/038c8d67-7cd3-48bf-a359-4ff2c577b32c-kube-api-access-ldhtf\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.173373 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.173273 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vf27\" (UniqueName: \"kubernetes.io/projected/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-kube-api-access-8vf27\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.173765 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.173741 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.206126 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.206099 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:27.461060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.460975 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:27.637757 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.637722 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx"] Nov 03 20:51:27.643071 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:51:27.643039 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod038c8d67_7cd3_48bf_a359_4ff2c577b32c.slice/crio-ec9fc930b223318c293c2b3f57405e153b139a4c8e0df2b83dcbc08a817c24c8 WatchSource:0}: Error finding container ec9fc930b223318c293c2b3f57405e153b139a4c8e0df2b83dcbc08a817c24c8: Status 404 returned error can't find the container with id ec9fc930b223318c293c2b3f57405e153b139a4c8e0df2b83dcbc08a817c24c8 Nov 03 20:51:27.645563 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.645527 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:51:27.645672 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.645606 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:51:27.645740 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.645693 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:51:27.837075 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.837033 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf"] Nov 03 20:51:27.842595 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:51:27.842567 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a56925_1c64_4165_abf5_3c9cc1cf22f3.slice/crio-72d83c5663c3d8141d315f40690e9fb08c3a4fdc90bd122d59d61db405121308 WatchSource:0}: Error finding container 72d83c5663c3d8141d315f40690e9fb08c3a4fdc90bd122d59d61db405121308: Status 404 returned error can't find the container with id 72d83c5663c3d8141d315f40690e9fb08c3a4fdc90bd122d59d61db405121308 Nov 03 20:51:27.845307 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.845270 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:51:27.845629 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.845380 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:51:27.845629 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.845424 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:51:27.992189 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.992140 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" event={"ID":"f1a56925-1c64-4165-abf5-3c9cc1cf22f3","Type":"ContainerStarted","Data":"e41c18a2406552fea87e4f7a67fe3c8ae6626b1c5014413781ba4eae7523091f"} Nov 03 20:51:27.992368 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.992255 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" event={"ID":"f1a56925-1c64-4165-abf5-3c9cc1cf22f3","Type":"ContainerStarted","Data":"72d83c5663c3d8141d315f40690e9fb08c3a4fdc90bd122d59d61db405121308"} Nov 03 20:51:27.994700 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.994672 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" event={"ID":"038c8d67-7cd3-48bf-a359-4ff2c577b32c","Type":"ContainerStarted","Data":"eeba465b1217eb955d9365e07d7f12c6f92682e2f4a6c26e814393a39ff78c52"} Nov 03 20:51:27.994853 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:27.994714 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" event={"ID":"038c8d67-7cd3-48bf-a359-4ff2c577b32c","Type":"ContainerStarted","Data":"ec9fc930b223318c293c2b3f57405e153b139a4c8e0df2b83dcbc08a817c24c8"} Nov 03 20:51:28.103523 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:28.103406 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" podStartSLOduration=2.103385774 podStartE2EDuration="2.103385774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:28.10127935 +0000 UTC m=+1359.659269887" watchObservedRunningTime="2026-11-03 20:51:28.103385774 +0000 UTC m=+1359.661376276" Nov 03 20:51:28.206421 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:28.206387 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:28.207884 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:28.207832 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:51:28.208043 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:28.207894 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" podUID="038c8d67-7cd3-48bf-a359-4ff2c577b32c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 20:51:28.210438 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:28.210395 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" podStartSLOduration=2.210381618 podStartE2EDuration="2.210381618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:28.209842636 +0000 UTC m=+1359.767833147" watchObservedRunningTime="2026-11-03 20:51:28.210381618 +0000 UTC m=+1359.768372117" Nov 03 20:51:28.461524 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:28.461444 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:28.462634 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:28.462609 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 20:51:28.462767 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:28.462662 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" podUID="f1a56925-1c64-4165-abf5-3c9cc1cf22f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 20:51:29.218359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:29.217511 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:51:29.218359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:29.217576 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" podUID="038c8d67-7cd3-48bf-a359-4ff2c577b32c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 20:51:29.462348 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:29.462307 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 20:51:29.462524 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:29.462384 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" podUID="f1a56925-1c64-4165-abf5-3c9cc1cf22f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 20:51:30.349359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:30.347459 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:51:30.349359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:30.347540 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" podUID="038c8d67-7cd3-48bf-a359-4ff2c577b32c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 20:51:30.479355 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:30.472853 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 20:51:30.479355 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:30.472951 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" podUID="f1a56925-1c64-4165-abf5-3c9cc1cf22f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 20:51:31.045424 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:31.042451 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:31.213359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:31.209416 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:51:31.213359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:31.209476 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" podUID="038c8d67-7cd3-48bf-a359-4ff2c577b32c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 20:51:31.264361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:31.257544 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:31.329418 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:31.328051 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:31.472361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:31.465804 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 20:51:31.472361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:31.465873 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" podUID="f1a56925-1c64-4165-abf5-3c9cc1cf22f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 20:51:31.623538 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:31.623429 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:31.624089 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:31.624050 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:51:31.690471 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:31.690424 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:31.774705 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:31.774664 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 20:51:31.838356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:31.837469 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:31.898609 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:31.898489 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:31.951467 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:31.951417 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:32.076559 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:32.076417 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:32.077109 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:32.077068 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:51:32.233421 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:32.183421 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:32.272359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:32.233761 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:51:32.344359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:32.330390 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" podUID="038c8d67-7cd3-48bf-a359-4ff2c577b32c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 20:51:32.344359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:32.332686 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:32.345178 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:32.345152 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:51:32.380360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:32.372096 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 20:51:32.391361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:32.385419 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:51:32.425358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:32.380463 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnnqh" podUID="524985f8-77ae-4383-bf88-355bde3b36ec" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.23:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:32.492416 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:32.490484 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 20:51:32.492416 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:32.490557 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" podUID="f1a56925-1c64-4165-abf5-3c9cc1cf22f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 20:51:32.636361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:32.631806 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:51:32.716047 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:32.716004 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:32.792728 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:32.792634 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:32.895361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:32.870119 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 03 20:51:33.006429 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:32.908631 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 03 20:51:33.033592 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:32.967500 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:33.109359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:33.103822 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:51:33.127558 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:33.127530 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:51:33.127860 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:33.127843 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:51:33.128041 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:33.128028 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:51:33.128205 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:33.128193 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:51:33.128544 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:33.128527 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:51:33.214937 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:33.214890 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:51:33.218439 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:33.218398 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" podUID="038c8d67-7cd3-48bf-a359-4ff2c577b32c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 20:51:33.371134 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:33.353746 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:33.454360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:33.385688 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 20:51:33.454802 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:33.426536 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:33.568382 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:33.501719 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 20:51:33.568382 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:33.501779 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" podUID="f1a56925-1c64-4165-abf5-3c9cc1cf22f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 20:51:33.650550 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:33.650442 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:33.729454 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:33.729394 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:33.811466 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:33.811424 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:34.098416 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:33.977473 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:51:34.098416 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:33.977557 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" podUID="57aeddbf-a41f-4cc0-8f76-d589898c2144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:34.098416 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:34.025716 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:34.098416 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:34.041046 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 03 20:51:34.098416 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:34.041805 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:34.143485 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:34.143445 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:51:34.143788 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:34.143755 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" podUID="fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:34.250361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:34.225506 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:51:34.250704 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:34.250663 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" podUID="038c8d67-7cd3-48bf-a359-4ff2c577b32c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 20:51:34.356447 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:34.355314 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:34.454984 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:34.454938 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 20:51:34.455611 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:34.455579 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Nov 03 20:51:34.455869 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:34.455845 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:34.520375 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:34.515162 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:51:34.576133 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:34.576097 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 20:51:34.580773 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:34.580738 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" podUID="f1a56925-1c64-4165-abf5-3c9cc1cf22f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 20:51:34.788449 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:34.776026 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:51:34.827396 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:34.819079 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:51:34.827396 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:34.822265 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:51:34.827396 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:34.822319 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:51:34.827396 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:34.822376 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:51:34.827396 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:34.822409 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:51:34.827396 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:34.822442 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:51:34.829238 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:34.829217 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:51:34.833411 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:34.833381 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:34.927283 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:34.927254 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:51:34.929325 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:34.929302 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:51:35.219361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:35.218694 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:51:35.219361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:35.218766 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" podUID="038c8d67-7cd3-48bf-a359-4ff2c577b32c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 20:51:35.461455 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:35.461422 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 20:51:35.461722 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:35.461685 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" podUID="f1a56925-1c64-4165-abf5-3c9cc1cf22f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 20:51:36.261365 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:36.256192 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:51:36.261365 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:36.256251 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" podUID="038c8d67-7cd3-48bf-a359-4ff2c577b32c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 20:51:36.461708 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:36.461670 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 20:51:36.461913 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:36.461723 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" podUID="f1a56925-1c64-4165-abf5-3c9cc1cf22f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 20:51:37.207132 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:37.207094 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:51:37.207326 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:37.207166 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" podUID="038c8d67-7cd3-48bf-a359-4ff2c577b32c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 20:51:37.462115 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:37.462014 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 20:51:37.462115 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:37.462079 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" podUID="f1a56925-1c64-4165-abf5-3c9cc1cf22f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 20:51:38.488601 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:38.488571 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:38.489122 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:38.489093 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:38.490016 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:38.489999 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 20:51:39.248537 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:39.248489 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:51:39.248825 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:39.248787 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" podUID="038c8d67-7cd3-48bf-a359-4ff2c577b32c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:39.383837 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:39.383800 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:39.384368 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:39.384301 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:39.385134 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:39.385115 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:51:51.746685 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:51:51.746641 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:51:52.999046 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:52.999011 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2"] Nov 03 20:51:53.003420 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.003393 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.103581 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.103543 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.103774 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.103618 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.103774 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.103692 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.103774 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.103764 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.103946 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.103829 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.103946 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.103908 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.104048 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.104009 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btl62\" (UniqueName: \"kubernetes.io/projected/1285885e-a374-4d7c-8109-0192cedd4fb2-kube-api-access-btl62\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.104105 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.104056 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.104105 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.104093 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1285885e-a374-4d7c-8109-0192cedd4fb2-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.104837 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.104814 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2"] Nov 03 20:51:53.205214 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.205181 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.205396 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.205223 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.205396 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.205298 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.205396 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.205382 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.205580 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.205485 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.205580 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.205518 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btl62\" (UniqueName: \"kubernetes.io/projected/1285885e-a374-4d7c-8109-0192cedd4fb2-kube-api-access-btl62\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.205580 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.205566 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.205733 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.205596 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1285885e-a374-4d7c-8109-0192cedd4fb2-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.205793 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.205777 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.205845 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.205789 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.205895 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.205852 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.205954 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.205896 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.206185 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.206128 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.206478 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.206448 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1285885e-a374-4d7c-8109-0192cedd4fb2-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.207806 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.207788 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.208659 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.208640 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.259503 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.259436 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btl62\" (UniqueName: \"kubernetes.io/projected/1285885e-a374-4d7c-8109-0192cedd4fb2-kube-api-access-btl62\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.259503 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.259441 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.317783 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.317675 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:53.742473 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.742441 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2"] Nov 03 20:51:53.746818 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:51:53.746790 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1285885e_a374_4d7c_8109_0192cedd4fb2.slice/crio-1a9df3b922ff1ee91db6b52ca34b1dd61be077888431483023f8505b1adb4922 WatchSource:0}: Error finding container 1a9df3b922ff1ee91db6b52ca34b1dd61be077888431483023f8505b1adb4922: Status 404 returned error can't find the container with id 1a9df3b922ff1ee91db6b52ca34b1dd61be077888431483023f8505b1adb4922 Nov 03 20:51:53.749291 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.749257 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:51:53.749400 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.749327 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:51:53.749400 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.749390 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:51:53.996011 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.995914 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" event={"ID":"1285885e-a374-4d7c-8109-0192cedd4fb2","Type":"ContainerStarted","Data":"2c3ff3ce94670ea937e47a72d79c495fb5e343a5e9b7e1d8d4f0bd38a8af7a0a"} Nov 03 20:51:53.996011 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:53.996006 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" event={"ID":"1285885e-a374-4d7c-8109-0192cedd4fb2","Type":"ContainerStarted","Data":"1a9df3b922ff1ee91db6b52ca34b1dd61be077888431483023f8505b1adb4922"} Nov 03 20:51:54.111516 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:54.111463 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" podStartSLOduration=2.111448866 podStartE2EDuration="2.111448866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:54.108897627 +0000 UTC m=+1385.666888146" watchObservedRunningTime="2026-11-03 20:51:54.111448866 +0000 UTC m=+1385.669439365" Nov 03 20:51:54.318665 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:54.318617 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:51:54.320255 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:54.320227 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 20:51:54.320420 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:54.320292 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" podUID="1285885e-a374-4d7c-8109-0192cedd4fb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 20:51:54.781552 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:54.781470 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz"] Nov 03 20:51:54.811914 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:54.811874 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:54.877478 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:54.877411 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz"] Nov 03 20:51:54.932127 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:54.932089 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:54.932358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:54.932149 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:54.932358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:54.932252 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:54.932358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:54.932278 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:54.932358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:54.932306 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:54.932611 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:54.932418 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:54.932611 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:54.932464 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:54.932611 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:54.932479 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgc5w\" (UniqueName: \"kubernetes.io/projected/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-kube-api-access-tgc5w\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:54.932611 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:54.932505 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:55.033750 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:55.033662 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:55.033750 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:55.033722 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:55.033972 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:55.033799 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:55.033972 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:55.033830 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:55.033972 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:55.033869 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:55.034143 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:55.034004 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:55.034143 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:55.034067 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:55.034143 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:55.034093 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgc5w\" (UniqueName: \"kubernetes.io/projected/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-kube-api-access-tgc5w\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:55.034143 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:55.034134 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:55.034425 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:55.034397 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:55.034578 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:55.034553 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:55.034660 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:55.034603 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:55.034799 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:55.034687 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:55.034912 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:55.034768 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:55.036704 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:55.036684 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:55.037040 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:55.037023 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:55.111407 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:55.111369 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgc5w\" (UniqueName: \"kubernetes.io/projected/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-kube-api-access-tgc5w\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:55.112809 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:55.112781 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:55.128557 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:55.128535 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:55.318175 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:55.318135 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 20:51:55.318418 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:55.318192 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" podUID="1285885e-a374-4d7c-8109-0192cedd4fb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 20:51:55.567837 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:55.567803 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz"] Nov 03 20:51:55.572803 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:51:55.572768 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e6937f_8195_42d2_a7e7_d14e98aba9cf.slice/crio-320a3ce5632e56254d522ad16a3d6cf1958a7c4bfcb0a58efbd3bb62efbb58af WatchSource:0}: Error finding container 320a3ce5632e56254d522ad16a3d6cf1958a7c4bfcb0a58efbd3bb62efbb58af: Status 404 returned error can't find the container with id 320a3ce5632e56254d522ad16a3d6cf1958a7c4bfcb0a58efbd3bb62efbb58af Nov 03 20:51:55.575741 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:55.575698 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:51:55.575856 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:55.575791 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:51:55.575856 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:55.575836 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:51:56.025005 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:56.024925 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" event={"ID":"b4e6937f-8195-42d2-a7e7-d14e98aba9cf","Type":"ContainerStarted","Data":"3b7e8d4ffa54a31130a2172c8f3963780d13fbbebb28321c295b883d50e04612"} Nov 03 20:51:56.025225 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:56.025196 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" event={"ID":"b4e6937f-8195-42d2-a7e7-d14e98aba9cf","Type":"ContainerStarted","Data":"320a3ce5632e56254d522ad16a3d6cf1958a7c4bfcb0a58efbd3bb62efbb58af"} Nov 03 20:51:56.129347 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:56.129295 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:51:56.131023 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:56.130996 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 20:51:56.131142 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:56.131044 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" podUID="b4e6937f-8195-42d2-a7e7-d14e98aba9cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 20:51:56.180174 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:56.180111 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" podStartSLOduration=2.180097092 podStartE2EDuration="2.180097092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:56.134139616 +0000 UTC m=+1387.692130143" watchObservedRunningTime="2026-11-03 20:51:56.180097092 +0000 UTC m=+1387.738087591" Nov 03 20:51:56.318023 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:56.317984 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 20:51:56.318195 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:56.318044 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" podUID="1285885e-a374-4d7c-8109-0192cedd4fb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 20:51:57.131987 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:57.131949 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 20:51:57.132670 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:57.132619 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" podUID="b4e6937f-8195-42d2-a7e7-d14e98aba9cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 20:51:57.319775 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:57.319738 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 20:51:57.321558 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:57.321524 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" podUID="1285885e-a374-4d7c-8109-0192cedd4fb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 20:51:58.133248 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:58.132792 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 20:51:58.133248 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:58.132856 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" podUID="b4e6937f-8195-42d2-a7e7-d14e98aba9cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 20:51:58.321359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:58.320710 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 20:51:58.321359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:58.320772 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" podUID="1285885e-a374-4d7c-8109-0192cedd4fb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 20:51:59.134726 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:59.134686 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 20:51:59.135212 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:59.134753 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" podUID="b4e6937f-8195-42d2-a7e7-d14e98aba9cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 20:51:59.327359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:59.326796 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 20:51:59.327359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:51:59.326861 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" podUID="1285885e-a374-4d7c-8109-0192cedd4fb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 20:52:00.129864 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:00.129819 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 20:52:00.130074 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:00.129880 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" podUID="b4e6937f-8195-42d2-a7e7-d14e98aba9cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 20:52:00.318768 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:00.318732 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 20:52:00.319373 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:00.319309 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" podUID="1285885e-a374-4d7c-8109-0192cedd4fb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 20:52:01.129153 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:01.129116 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 20:52:01.129357 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:01.129175 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" podUID="b4e6937f-8195-42d2-a7e7-d14e98aba9cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 20:52:01.399780 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:01.399695 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:52:01.401879 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:01.401858 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:52:01.409470 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:01.409215 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:52:02.129637 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:02.129604 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 20:52:02.129810 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:02.129670 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" podUID="b4e6937f-8195-42d2-a7e7-d14e98aba9cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 20:52:03.241620 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:03.241589 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:52:03.242616 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:03.242592 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:52:03.243369 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:03.243351 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:52:07.057359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:07.050430 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:07.371741 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:07.371625 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:07.504486 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:07.504439 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:07.513535 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:07.513486 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:07.684537 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:07.684428 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:07.738801 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:07.738724 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:07.792548 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:07.792487 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:07.845698 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:07.845656 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:08.036479 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:08.036376 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:08.051285 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:08.051247 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 03 20:52:08.051942 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:08.051919 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:52:08.072573 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:08.072529 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:08.400363 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:08.374423 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:08.400363 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:08.375206 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:52:08.512616 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:08.512557 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:08.513074 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:08.513039 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:52:08.545471 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:08.528429 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:08.695733 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:08.695627 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:08.726052 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:08.697951 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:08.753378 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:08.753286 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:52:08.825433 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:08.801423 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:08.938371 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:08.925279 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 20:52:09.042544 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:09.042440 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:09.088388 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:09.065124 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 03 20:52:09.090866 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:09.090824 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:52:09.141627 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:09.117142 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:52:09.306635 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:09.306609 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:52:09.352379 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:09.333259 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:09.366233 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:09.359821 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:52:09.366233 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:09.359912 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:52:09.366233 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:09.359954 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:52:09.396865 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:09.388430 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:09.519881 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:09.519829 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:09.593793 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:09.593689 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:52:09.594065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:09.594035 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" podUID="f1a56925-1c64-4165-abf5-3c9cc1cf22f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:10.048420 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:10.043601 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 20:52:10.048420 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:10.047438 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:10.094083 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:10.093183 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:52:10.128804 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:10.128760 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:10.137392 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:10.128810 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:52:10.164887 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:10.128840 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:10.165372 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:10.137975 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 03 20:52:10.395412 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:10.390226 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:10.602356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:10.583196 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 20:52:11.059280 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:11.048453 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:11.169374 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:11.163918 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:11.197381 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:11.178406 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:11.197381 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:11.192495 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:11.197381 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:11.192988 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:52:11.197911 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:11.197884 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:11.297752 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:11.297708 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:11.463128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:11.463037 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:52:11.463718 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:11.463525 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:52:11.464131 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:11.464118 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:52:11.464727 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:11.464707 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:52:11.464911 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:11.464899 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:52:11.465040 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:11.465029 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:52:11.465443 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:11.465420 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:52:11.465661 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:11.465646 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:52:11.466077 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:11.466029 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:52:21.744695 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:52:21.744659 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:52:24.999209 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:24.999173 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7"] Nov 03 20:52:25.003564 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.003546 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.046666 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.046641 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq"] Nov 03 20:52:25.050527 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.050508 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.081465 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.081440 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc"] Nov 03 20:52:25.085280 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.085265 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.095123 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.095100 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7"] Nov 03 20:52:25.138214 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.138179 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq"] Nov 03 20:52:25.148039 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.148010 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.148196 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.148065 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.148196 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.148187 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndb6g\" (UniqueName: \"kubernetes.io/projected/f62e8863-2c8c-4658-8ecf-645709ec29e2-kube-api-access-ndb6g\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.148273 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.148214 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.148315 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.148301 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.148387 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.148327 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.148437 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.148424 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.148497 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.148484 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.148569 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.148546 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f62e8863-2c8c-4658-8ecf-645709ec29e2-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.148606 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.148589 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68r5x\" (UniqueName: \"kubernetes.io/projected/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-kube-api-access-68r5x\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.148641 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.148624 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.148682 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.148665 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.148716 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.148699 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.148752 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.148739 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.148785 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.148768 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.148820 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.148786 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.148963 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.148951 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.148996 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.148978 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.187162 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.187132 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc"] Nov 03 20:52:25.250769 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.250695 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndb6g\" (UniqueName: \"kubernetes.io/projected/f62e8863-2c8c-4658-8ecf-645709ec29e2-kube-api-access-ndb6g\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.250769 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.250732 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.250769 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.250767 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.251043 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.250790 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.251043 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.250828 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.251043 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.250861 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnjrm\" (UniqueName: \"kubernetes.io/projected/adf4f7e5-e60d-484e-823f-1a40134f48fc-kube-api-access-bnjrm\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.251043 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.250886 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.251043 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.250922 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.251043 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.250962 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f62e8863-2c8c-4658-8ecf-645709ec29e2-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.251384 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.251243 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.251384 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.251274 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.251384 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.251309 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.251384 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.251370 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.251780 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.251753 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68r5x\" (UniqueName: \"kubernetes.io/projected/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-kube-api-access-68r5x\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.251780 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.251767 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f62e8863-2c8c-4658-8ecf-645709ec29e2-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.251961 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.251823 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.251961 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.251844 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.251961 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.251868 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.251961 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.251893 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.252164 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.251992 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adf4f7e5-e60d-484e-823f-1a40134f48fc-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.252164 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.252040 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.252164 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.252130 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.252164 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.252139 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.252387 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.252175 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.252387 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.252212 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.252387 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.252277 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.252387 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.252317 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.252582 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.252402 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.252582 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.252455 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.252582 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.252497 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.252582 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.252556 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.252781 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.252629 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.252781 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.252634 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.252781 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.252658 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.252781 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.252685 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.252781 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.252694 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.253019 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.252859 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.253973 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.253945 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.254924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.254905 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.255029 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.255012 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.255474 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.255454 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.305409 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.305376 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndb6g\" (UniqueName: \"kubernetes.io/projected/f62e8863-2c8c-4658-8ecf-645709ec29e2-kube-api-access-ndb6g\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.305645 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.305626 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.306403 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.306387 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.306456 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.306443 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68r5x\" (UniqueName: \"kubernetes.io/projected/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-kube-api-access-68r5x\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.317461 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.317438 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:25.354269 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.354240 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.354415 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.354279 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.354415 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.354382 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnjrm\" (UniqueName: \"kubernetes.io/projected/adf4f7e5-e60d-484e-823f-1a40134f48fc-kube-api-access-bnjrm\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.354505 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.354419 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.354559 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.354508 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.354620 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.354564 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adf4f7e5-e60d-484e-823f-1a40134f48fc-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.354673 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.354640 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.354729 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.354690 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.354729 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.354696 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.354831 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.354740 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.355096 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.355074 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.355217 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.355197 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.355391 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.355326 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.355447 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.355395 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adf4f7e5-e60d-484e-823f-1a40134f48fc-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.357381 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.357316 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.357480 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.357444 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.364177 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.364158 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:25.436627 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.436586 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.438932 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.438906 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnjrm\" (UniqueName: \"kubernetes.io/projected/adf4f7e5-e60d-484e-823f-1a40134f48fc-kube-api-access-bnjrm\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.699880 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.699848 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:25.752189 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.752154 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7"] Nov 03 20:52:25.800309 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:52:25.800266 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf62e8863_2c8c_4658_8ecf_645709ec29e2.slice/crio-9cdbd3fa0a52884de72616a21a12160347190d46d77591c7a32a94f76a423118 WatchSource:0}: Error finding container 9cdbd3fa0a52884de72616a21a12160347190d46d77591c7a32a94f76a423118: Status 404 returned error can't find the container with id 9cdbd3fa0a52884de72616a21a12160347190d46d77591c7a32a94f76a423118 Nov 03 20:52:25.803859 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.803815 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:52:25.803989 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.803918 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:52:25.803989 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.803963 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:52:25.807841 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.807774 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq"] Nov 03 20:52:25.814985 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:52:25.814959 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod353543b3_c2cc_4563_a2a0_ddcf5b03db1b.slice/crio-ba174acdf2f4730b2ca0835ac86301d6710e757a9c08628f8b34c3f93dc77410 WatchSource:0}: Error finding container ba174acdf2f4730b2ca0835ac86301d6710e757a9c08628f8b34c3f93dc77410: Status 404 returned error can't find the container with id ba174acdf2f4730b2ca0835ac86301d6710e757a9c08628f8b34c3f93dc77410 Nov 03 20:52:25.817532 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.817505 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:52:25.817640 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.817576 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:52:25.817640 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:25.817617 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:52:26.082648 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:26.082610 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc"] Nov 03 20:52:26.089428 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:52:26.089399 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf4f7e5_e60d_484e_823f_1a40134f48fc.slice/crio-3f5aebbcc16bc934bd23da2036e80a3cfc5c789bb2d81314daf55db910376436 WatchSource:0}: Error finding container 3f5aebbcc16bc934bd23da2036e80a3cfc5c789bb2d81314daf55db910376436: Status 404 returned error can't find the container with id 3f5aebbcc16bc934bd23da2036e80a3cfc5c789bb2d81314daf55db910376436 Nov 03 20:52:26.092928 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:26.092887 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:52:26.093011 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:26.092978 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:52:26.093048 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:26.093021 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:52:26.583062 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:26.580833 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" event={"ID":"353543b3-c2cc-4563-a2a0-ddcf5b03db1b","Type":"ContainerStarted","Data":"267268c7a56d422d88a164baa20584df3662c33b63e54ac87e30ba71e43b08f4"} Nov 03 20:52:26.583062 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:26.580892 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" event={"ID":"353543b3-c2cc-4563-a2a0-ddcf5b03db1b","Type":"ContainerStarted","Data":"ba174acdf2f4730b2ca0835ac86301d6710e757a9c08628f8b34c3f93dc77410"} Nov 03 20:52:26.583270 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:26.583189 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" event={"ID":"f62e8863-2c8c-4658-8ecf-645709ec29e2","Type":"ContainerStarted","Data":"9b2f630a7f9765da0111ba5fab6da949cad3038109aec67fad74464c5d9bddef"} Nov 03 20:52:26.583325 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:26.583274 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" event={"ID":"f62e8863-2c8c-4658-8ecf-645709ec29e2","Type":"ContainerStarted","Data":"9cdbd3fa0a52884de72616a21a12160347190d46d77591c7a32a94f76a423118"} Nov 03 20:52:26.648403 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:26.638615 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" event={"ID":"adf4f7e5-e60d-484e-823f-1a40134f48fc","Type":"ContainerStarted","Data":"6582c21ef2abd9d2ddab5c8f0f227b8a2355fbcb1cdea4d67c905f91f33452ea"} Nov 03 20:52:26.648403 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:26.638680 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" event={"ID":"adf4f7e5-e60d-484e-823f-1a40134f48fc","Type":"ContainerStarted","Data":"3f5aebbcc16bc934bd23da2036e80a3cfc5c789bb2d81314daf55db910376436"} Nov 03 20:52:26.720358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:26.712025 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:26.921735 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:26.913669 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" podStartSLOduration=2.913644197 podStartE2EDuration="2.913644197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:26.735519563 +0000 UTC m=+1418.293510103" watchObservedRunningTime="2026-11-03 20:52:26.913644197 +0000 UTC m=+1418.471634696" Nov 03 20:52:26.921735 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:26.914634 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" podStartSLOduration=1.9146140630000001 podStartE2EDuration="1.914614063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:26.894035016 +0000 UTC m=+1418.452025512" watchObservedRunningTime="2026-11-03 20:52:26.914614063 +0000 UTC m=+1418.472604573" Nov 03 20:52:27.049360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.039935 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" podStartSLOduration=3.039913163 podStartE2EDuration="3.039913163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:27.037863709 +0000 UTC m=+1418.595854227" watchObservedRunningTime="2026-11-03 20:52:27.039913163 +0000 UTC m=+1418.597903664" Nov 03 20:52:27.327360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.326650 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:27.327881 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.327599 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:52:27.327881 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.327669 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" podUID="f62e8863-2c8c-4658-8ecf-645709ec29e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 20:52:27.356528 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.352674 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8"] Nov 03 20:52:27.375366 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.363633 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.375366 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.375348 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.375600 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.375513 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a13da4c7-1a94-4992-b4de-b88846e62f21-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.396376 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.376711 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.396376 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.376776 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnv2s\" (UniqueName: \"kubernetes.io/projected/a13da4c7-1a94-4992-b4de-b88846e62f21-kube-api-access-rnv2s\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.396376 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.376807 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.396376 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.376843 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.396376 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.376891 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.396376 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.376982 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.396376 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.377056 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.396376 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.377359 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:27.396376 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.384058 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8"] Nov 03 20:52:27.417463 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.397943 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 20:52:27.417463 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.398001 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" podUID="353543b3-c2cc-4563-a2a0-ddcf5b03db1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 20:52:27.483080 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.478291 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.483080 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.478349 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnv2s\" (UniqueName: \"kubernetes.io/projected/a13da4c7-1a94-4992-b4de-b88846e62f21-kube-api-access-rnv2s\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.483080 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.478369 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.483080 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.478390 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.483080 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.478416 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.483080 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.478463 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.483080 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.478502 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.483080 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.478567 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.483080 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.478647 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a13da4c7-1a94-4992-b4de-b88846e62f21-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.483080 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.479733 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a13da4c7-1a94-4992-b4de-b88846e62f21-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.483080 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.483054 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.544532 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.483453 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.544532 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.483698 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.544532 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.491687 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.544532 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.491941 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.563169 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.563132 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.569602 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.569570 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.577785 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.574831 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnv2s\" (UniqueName: \"kubernetes.io/projected/a13da4c7-1a94-4992-b4de-b88846e62f21-kube-api-access-rnv2s\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:27.705548 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.705512 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:52:27.711478 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.711432 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" podUID="adf4f7e5-e60d-484e-823f-1a40134f48fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 20:52:27.871856 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:27.871763 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:28.365945 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:28.343735 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:52:28.365945 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:28.343797 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" podUID="f62e8863-2c8c-4658-8ecf-645709ec29e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 20:52:28.365945 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:28.364720 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 20:52:28.365945 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:28.364774 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" podUID="353543b3-c2cc-4563-a2a0-ddcf5b03db1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 20:52:28.467830 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:28.467603 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8"] Nov 03 20:52:28.470796 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:28.470759 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:52:28.470883 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:28.470843 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:52:28.470920 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:28.470882 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:52:28.707561 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:28.707526 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" event={"ID":"a13da4c7-1a94-4992-b4de-b88846e62f21","Type":"ContainerStarted","Data":"04a7d4910f7703e76f12585f1fb088147c87a8c3a12ac9b48c285e82aad9d399"} Nov 03 20:52:28.730398 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:28.730368 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:52:28.730661 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:28.730631 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" podUID="adf4f7e5-e60d-484e-823f-1a40134f48fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 20:52:29.318359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:29.317991 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:52:29.318359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:29.318064 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" podUID="f62e8863-2c8c-4658-8ecf-645709ec29e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 20:52:29.393361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:29.370190 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 20:52:29.393361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:29.370252 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" podUID="353543b3-c2cc-4563-a2a0-ddcf5b03db1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 20:52:29.704418 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:29.701709 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:52:29.704418 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:29.701777 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" podUID="adf4f7e5-e60d-484e-823f-1a40134f48fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 20:52:29.745382 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:29.745243 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" event={"ID":"a13da4c7-1a94-4992-b4de-b88846e62f21","Type":"ContainerStarted","Data":"da09b39a2849891ace3dc3df19a1c5cabc72cc3f5f307bf6982cfa097ccd4b02"} Nov 03 20:52:29.884358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:29.877302 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:29.884358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:29.877452 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:52:29.884358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:29.877505 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" podUID="a13da4c7-1a94-4992-b4de-b88846e62f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 20:52:30.329361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:30.323911 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:52:30.329361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:30.323970 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" podUID="f62e8863-2c8c-4658-8ecf-645709ec29e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 20:52:30.365355 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:30.364730 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 20:52:30.365355 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:30.364791 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" podUID="353543b3-c2cc-4563-a2a0-ddcf5b03db1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 20:52:30.705415 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:30.703867 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:52:30.705415 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:30.703931 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" podUID="adf4f7e5-e60d-484e-823f-1a40134f48fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 20:52:30.887987 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:30.887947 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:52:30.888371 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:30.888288 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" podUID="a13da4c7-1a94-4992-b4de-b88846e62f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 20:52:31.174467 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:31.174416 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:31.327360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:31.185426 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 20:52:31.356595 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:31.212420 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:31.357367 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:31.212938 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:31.357796 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:31.212976 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:31.358289 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:31.328104 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:52:31.358289 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:31.358228 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" podUID="f62e8863-2c8c-4658-8ecf-645709ec29e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 20:52:31.382357 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:31.380599 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 20:52:31.382357 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:31.380654 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" podUID="353543b3-c2cc-4563-a2a0-ddcf5b03db1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 20:52:31.468590 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:31.468477 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:31.468951 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:31.468920 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:31.605739 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:31.605694 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:31.632661 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:31.632619 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:31.711361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:31.707849 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:52:31.711361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:31.707913 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" podUID="adf4f7e5-e60d-484e-823f-1a40134f48fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 20:52:31.877303 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:31.877178 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:52:31.877782 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:31.877638 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" podUID="a13da4c7-1a94-4992-b4de-b88846e62f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 20:52:32.269432 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.262050 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:32.269432 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.262137 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:32.269432 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.262440 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:52:32.320471 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.320011 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:52:32.320471 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.320067 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" podUID="f62e8863-2c8c-4658-8ecf-645709ec29e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 20:52:32.354488 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.354439 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnnqh" podUID="524985f8-77ae-4383-bf88-355bde3b36ec" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.23:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:32.413128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.413085 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:32.413815 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.413791 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:52:32.417290 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.417259 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:32.426115 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.426083 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:32.427579 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.426488 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:32.454623 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.439592 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:52:32.454623 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.452006 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:52:32.454623 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.439989 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 20:52:32.454623 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.452230 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" podUID="353543b3-c2cc-4563-a2a0-ddcf5b03db1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 20:52:32.454623 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.440029 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:32.454623 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.452660 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:52:32.477185 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.476758 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:32.503141 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.484143 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 03 20:52:32.503141 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.489314 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:52:32.503141 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.489403 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:52:32.563770 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.531927 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" podStartSLOduration=5.531904122 podStartE2EDuration="5.531904122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:29.875756459 +0000 UTC m=+1421.433746982" watchObservedRunningTime="2026-11-03 20:52:32.531904122 +0000 UTC m=+1424.089894623" Nov 03 20:52:32.573456 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.562942 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:52:32.641958 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.640995 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:32.650875 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.650845 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:52:32.707632 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.707599 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:52:32.717619 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.717580 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" podUID="adf4f7e5-e60d-484e-823f-1a40134f48fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 20:52:32.815043 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.813502 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:52:32.875540 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.873774 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:52:32.875540 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.873833 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" podUID="a13da4c7-1a94-4992-b4de-b88846e62f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 20:52:32.876010 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.875969 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:52:32.972868 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:32.972841 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:52:33.050794 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:33.050757 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:52:33.051590 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:33.051563 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:52:33.054304 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:33.054268 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:52:33.125064 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:33.124987 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:52:33.334962 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:33.334931 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:52:33.335223 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:33.335194 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" podUID="f62e8863-2c8c-4658-8ecf-645709ec29e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 20:52:33.365548 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:33.365522 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 20:52:33.365753 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:33.365734 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" podUID="353543b3-c2cc-4563-a2a0-ddcf5b03db1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 20:52:33.720722 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:33.717855 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:52:33.723852 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:33.721754 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" podUID="adf4f7e5-e60d-484e-823f-1a40134f48fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 20:52:33.873510 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:33.872496 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:52:33.873510 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:33.872557 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" podUID="a13da4c7-1a94-4992-b4de-b88846e62f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 20:52:34.572747 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:34.572718 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:34.575151 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:34.574730 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:34.580732 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:34.580696 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:52:34.638795 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:34.635648 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:34.638795 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:34.637006 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:34.638795 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:34.638519 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:52:34.872750 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:34.872663 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:52:34.873739 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:34.873304 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" podUID="a13da4c7-1a94-4992-b4de-b88846e62f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 20:52:34.941241 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:34.941212 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:34.942114 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:34.942089 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:34.943879 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:34.943858 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:52:35.872411 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:35.872379 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:52:35.872590 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:35.872428 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" podUID="a13da4c7-1a94-4992-b4de-b88846e62f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 20:52:37.113857 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:37.113807 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:37.114423 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:37.114372 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:37.115838 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:37.115818 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 20:52:38.199203 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:38.199170 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l"] Nov 03 20:52:38.199734 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:38.199376 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" podUID="a3f1f367-79f6-4068-90bc-6373d5651ab4" containerName="istio-proxy" containerID="cri-o://a7995bb6789c04d6541cbcc96ca030fec43b65b830e669181976a0716a2a24a3" gracePeriod=30 Nov 03 20:52:38.246558 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:38.246525 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4"] Nov 03 20:52:38.246768 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:38.246741 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" podUID="d07d8d19-5ed5-4f0b-9084-390494a695c6" containerName="istio-proxy" containerID="cri-o://d03028ce68d694afe8f787a42a1b19eb91ab15cdc8462cba5e3cd7a729ba77a7" gracePeriod=30 Nov 03 20:52:38.253493 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:38.253464 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc"] Nov 03 20:52:38.253703 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:38.253680 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" podUID="c4f6d970-a5d4-4d9d-89ad-495365b7d652" containerName="istio-proxy" containerID="cri-o://21dae403d96d60c6909eacbc1fe3b288d73f64353f0059bd83fb18fc08193bd3" gracePeriod=30 Nov 03 20:52:38.293557 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:38.293528 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8"] Nov 03 20:52:38.293782 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:38.293760 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" podUID="ff0d6342-c880-4f83-89e5-2b5e14d512ee" containerName="istio-proxy" containerID="cri-o://f1dc2286ce1c2afe2160af159cad5e1cc85a5ec67d65973346ac0e9aeff3cf5c" gracePeriod=30 Nov 03 20:52:39.595377 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.595321 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2"] Nov 03 20:52:39.599933 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.599915 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.659523 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.659464 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2"] Nov 03 20:52:39.699589 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.699558 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8170541-ea8d-400b-9636-52a81628985f-istio-podinfo\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.699758 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.699619 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8170541-ea8d-400b-9636-52a81628985f-istio-token\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.699805 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.699758 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-istio-envoy\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.699849 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.699807 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-istio-data\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.699980 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.699959 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-workload-socket\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.700070 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.700044 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlxhf\" (UniqueName: \"kubernetes.io/projected/d8170541-ea8d-400b-9636-52a81628985f-kube-api-access-mlxhf\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.700166 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.700151 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-e20715ef-gw-openshift-default-dockercfg-9hvwr\"" Nov 03 20:52:39.700247 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.700229 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-workload-certs\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.700286 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.700272 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-credential-socket\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.700378 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.700364 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8170541-ea8d-400b-9636-52a81628985f-istiod-ca-cert\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.801930 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.801898 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-workload-certs\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.802077 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.801947 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-credential-socket\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.802077 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.802001 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8170541-ea8d-400b-9636-52a81628985f-istiod-ca-cert\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.802216 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.802093 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8170541-ea8d-400b-9636-52a81628985f-istio-podinfo\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.802216 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.802140 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8170541-ea8d-400b-9636-52a81628985f-istio-token\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.802301 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.802230 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-istio-envoy\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.802374 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.802307 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-istio-data\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.802439 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.802368 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-workload-socket\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.802439 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.802410 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlxhf\" (UniqueName: \"kubernetes.io/projected/d8170541-ea8d-400b-9636-52a81628985f-kube-api-access-mlxhf\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.802439 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.802432 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-workload-certs\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.802594 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.802530 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-credential-socket\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.802801 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.802774 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-workload-socket\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.802902 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.802876 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-istio-data\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.802902 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.802895 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8170541-ea8d-400b-9636-52a81628985f-istiod-ca-cert\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.804841 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.804803 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8170541-ea8d-400b-9636-52a81628985f-istio-podinfo\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.804930 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.804819 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-istio-envoy\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.872345 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.872244 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlxhf\" (UniqueName: \"kubernetes.io/projected/d8170541-ea8d-400b-9636-52a81628985f-kube-api-access-mlxhf\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.872578 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.872558 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8170541-ea8d-400b-9636-52a81628985f-istio-token\") pod \"e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:39.913953 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:39.913915 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:40.354374 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:40.354322 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2"] Nov 03 20:52:40.358661 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:52:40.358631 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8170541_ea8d_400b_9636_52a81628985f.slice/crio-df8a9075c3b261e34bca1ad8e874f2350629c7347b7f753eefb0bc10c2b4edaf WatchSource:0}: Error finding container df8a9075c3b261e34bca1ad8e874f2350629c7347b7f753eefb0bc10c2b4edaf: Status 404 returned error can't find the container with id df8a9075c3b261e34bca1ad8e874f2350629c7347b7f753eefb0bc10c2b4edaf Nov 03 20:52:40.361110 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:40.361079 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:52:40.361210 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:40.361145 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:52:40.361210 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:40.361172 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:52:41.174104 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:41.174066 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" event={"ID":"d8170541-ea8d-400b-9636-52a81628985f","Type":"ContainerStarted","Data":"39298848a37ab689208b4dc15abf27c2ecad7c612df8a1413fbb1a83f6a7684e"} Nov 03 20:52:41.174547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:41.174126 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" event={"ID":"d8170541-ea8d-400b-9636-52a81628985f","Type":"ContainerStarted","Data":"df8a9075c3b261e34bca1ad8e874f2350629c7347b7f753eefb0bc10c2b4edaf"} Nov 03 20:52:41.380223 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:41.380170 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" podStartSLOduration=2.380153719 podStartE2EDuration="2.380153719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:41.332719396 +0000 UTC m=+1432.890709906" watchObservedRunningTime="2026-11-03 20:52:41.380153719 +0000 UTC m=+1432.938144217" Nov 03 20:52:41.914191 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:41.914158 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:41.919721 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:41.919701 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:42.029748 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:42.029707 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-9pl8l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:52:42.029903 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:42.029767 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" podUID="a3f1f367-79f6-4068-90bc-6373d5651ab4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:52:42.181887 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:42.181767 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:42.183296 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:42.183262 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:52:44.143356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:44.138017 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:44.342360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:44.340918 4493 generic.go:356] "Generic (PLEG): container finished" podID="a3f1f367-79f6-4068-90bc-6373d5651ab4" containerID="a7995bb6789c04d6541cbcc96ca030fec43b65b830e669181976a0716a2a24a3" exitCode=0 Nov 03 20:52:44.342360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:44.341017 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" event={"ID":"a3f1f367-79f6-4068-90bc-6373d5651ab4","Type":"ContainerDied","Data":"a7995bb6789c04d6541cbcc96ca030fec43b65b830e669181976a0716a2a24a3"} Nov 03 20:52:44.369360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:44.369292 4493 generic.go:356] "Generic (PLEG): container finished" podID="d07d8d19-5ed5-4f0b-9084-390494a695c6" containerID="d03028ce68d694afe8f787a42a1b19eb91ab15cdc8462cba5e3cd7a729ba77a7" exitCode=0 Nov 03 20:52:44.369564 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:44.369475 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" event={"ID":"d07d8d19-5ed5-4f0b-9084-390494a695c6","Type":"ContainerDied","Data":"d03028ce68d694afe8f787a42a1b19eb91ab15cdc8462cba5e3cd7a729ba77a7"} Nov 03 20:52:44.377358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:44.375893 4493 generic.go:356] "Generic (PLEG): container finished" podID="c4f6d970-a5d4-4d9d-89ad-495365b7d652" containerID="21dae403d96d60c6909eacbc1fe3b288d73f64353f0059bd83fb18fc08193bd3" exitCode=0 Nov 03 20:52:44.377358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:44.375988 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" event={"ID":"c4f6d970-a5d4-4d9d-89ad-495365b7d652","Type":"ContainerDied","Data":"21dae403d96d60c6909eacbc1fe3b288d73f64353f0059bd83fb18fc08193bd3"} Nov 03 20:52:44.395414 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:44.393601 4493 generic.go:356] "Generic (PLEG): container finished" podID="ff0d6342-c880-4f83-89e5-2b5e14d512ee" containerID="f1dc2286ce1c2afe2160af159cad5e1cc85a5ec67d65973346ac0e9aeff3cf5c" exitCode=0 Nov 03 20:52:44.395414 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:44.394689 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" event={"ID":"ff0d6342-c880-4f83-89e5-2b5e14d512ee","Type":"ContainerDied","Data":"f1dc2286ce1c2afe2160af159cad5e1cc85a5ec67d65973346ac0e9aeff3cf5c"} Nov 03 20:52:44.434539 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:44.434479 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:44.445469 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:44.445419 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Nov 03 20:52:44.586661 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:44.586621 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:44.631501 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:44.631461 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:52:44.631808 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:44.631774 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" podUID="a3fb9508-37d2-47fa-bb02-ae7735c9f408" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:44.760540 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:44.760430 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:44.808476 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:44.808428 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:44.880358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:44.878422 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:44.977165 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:44.976424 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:52:45.057019 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:45.056953 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:45.057467 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:45.057425 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:45.123283 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:45.123242 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:45.144490 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:45.144436 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:45.145555 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:45.145530 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:52:45.540456 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:45.507533 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 03 20:52:45.540456 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:45.508163 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:52:45.633366 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:45.609703 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 03 20:52:45.633366 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:45.610162 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:52:45.689446 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:45.689386 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:45.808433 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:45.796693 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-85pk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 20:52:45.808433 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:45.796774 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" podUID="ff0d6342-c880-4f83-89e5-2b5e14d512ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 20:52:45.808433 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:45.796906 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-5qqpc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 20:52:45.808433 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:45.796929 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" podUID="c4f6d970-a5d4-4d9d-89ad-495365b7d652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 20:52:45.857363 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:45.831531 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:52:45.857363 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:45.832725 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:52:45.909041 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:45.890407 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:45.909041 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:45.890543 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 03 20:52:45.909041 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:45.890841 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:52:45.992361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:45.988835 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:46.018363 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:46.018146 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:52:46.069421 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:46.059842 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:46.078414 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:46.077616 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:52:46.113486 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:46.113443 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:46.118504 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:46.118474 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:52:46.147414 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:46.135647 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:46.148592 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:46.148554 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:52:46.222363 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:46.153251 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 20:52:46.779360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:46.772207 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 03 20:52:46.918364 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:46.914728 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:47.164359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:46.934661 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:52:47.164359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:46.990621 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:47.254400 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:47.033764 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 03 20:52:47.352475 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:47.317176 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 20:52:47.489419 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:47.472197 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:47.570311 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:47.570262 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnnqh" podUID="524985f8-77ae-4383-bf88-355bde3b36ec" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.23:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:47.601469 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:47.593104 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:52:47.627385 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:47.600065 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 20:52:47.845954 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:47.844270 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:47.846363 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:47.846297 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:52:47.871242 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:47.871199 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" podUID="1285885e-a374-4d7c-8109-0192cedd4fb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:48.203562 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:48.203455 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:48.217510 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:48.217444 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 03 20:52:48.244782 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:48.240420 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:48.245487 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:48.221425 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:52:48.378442 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:48.365606 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:48.515005 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:48.514946 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:48.628764 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:48.628706 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 20:52:48.650975 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:48.650924 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:52:48.691042 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:48.690995 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 20:52:48.895894 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:48.871677 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 03 20:52:48.953371 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:48.949721 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-dbbg4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:52:48.953371 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:48.949781 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" podUID="d07d8d19-5ed5-4f0b-9084-390494a695c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:48.953371 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:48.949856 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded" Nov 03 20:52:49.001368 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:52:48.997569 4493 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.658s" Nov 03 20:52:49.030362 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:49.028460 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:52:49.034363 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:49.032421 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" podUID="57aeddbf-a41f-4cc0-8f76-d589898c2144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:49.074519 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:49.074481 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:52:49.074794 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:49.074766 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" podUID="fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:49.281441 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:49.276165 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:49.374685 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:49.374635 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 03 20:52:49.458929 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:49.378251 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 03 20:52:49.459418 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:49.446953 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:52:49.584364 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:49.572973 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:52:49.963647 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:49.963561 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" podUID="b4e6937f-8195-42d2-a7e7-d14e98aba9cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:49.963871 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:49.659431 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 20:52:49.964913 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:49.650628 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:52:49.965383 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:49.917935 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 20:52:49.966379 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:49.946960 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:52:50.002730 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.002691 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 03 20:52:50.008616 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.008569 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:52:50.014367 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.010726 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:50.088522 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.088489 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff0d6342-c880-4f83-89e5-2b5e14d512ee-kube-api-access-qs98q\" (UniqueName: \"kubernetes.io/projected/ff0d6342-c880-4f83-89e5-2b5e14d512ee-kube-api-access-qs98q\") pod \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " Nov 03 20:52:50.088903 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.088881 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-data\") pod \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " Nov 03 20:52:50.106672 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.106637 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-token\" (UniqueName: \"kubernetes.io/projected/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-token\") pod \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " Nov 03 20:52:50.107089 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.107069 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istiod-ca-cert\") pod \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " Nov 03 20:52:50.107314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.107298 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-credential-socket\") pod \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " Nov 03 20:52:50.107652 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.107622 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-podinfo\") pod \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " Nov 03 20:52:50.107874 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.107852 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-workload-socket\") pod \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " Nov 03 20:52:50.108079 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.108062 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-workload-certs\") pod \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " Nov 03 20:52:50.108225 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.108210 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-envoy\") pod \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\" (UID: \"ff0d6342-c880-4f83-89e5-2b5e14d512ee\") " Nov 03 20:52:50.108667 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.108645 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-data" pod "ff0d6342-c880-4f83-89e5-2b5e14d512ee" (UID: "ff0d6342-c880-4f83-89e5-2b5e14d512ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:50.108995 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.108974 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-credential-socket" pod "ff0d6342-c880-4f83-89e5-2b5e14d512ee" (UID: "ff0d6342-c880-4f83-89e5-2b5e14d512ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:50.109607 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.109584 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-workload-socket" pod "ff0d6342-c880-4f83-89e5-2b5e14d512ee" (UID: "ff0d6342-c880-4f83-89e5-2b5e14d512ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:50.110302 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.110279 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istiod-ca-cert" pod "ff0d6342-c880-4f83-89e5-2b5e14d512ee" (UID: "ff0d6342-c880-4f83-89e5-2b5e14d512ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:50.110714 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.110692 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-workload-certs" pod "ff0d6342-c880-4f83-89e5-2b5e14d512ee" (UID: "ff0d6342-c880-4f83-89e5-2b5e14d512ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:50.363378 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.363350 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:50.363982 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.363961 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:50.364138 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.364123 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:50.364263 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.364250 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:50.364415 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.364401 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:50.417360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.395063 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" event={"ID":"a3f1f367-79f6-4068-90bc-6373d5651ab4","Type":"ContainerDied","Data":"60f169b62aae21b58c1656d8f42bb5eef8f43b6e78e8d350dc2f39035b5dc1f0"} Nov 03 20:52:50.417360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.395129 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60f169b62aae21b58c1656d8f42bb5eef8f43b6e78e8d350dc2f39035b5dc1f0" Nov 03 20:52:50.430740 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.430704 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-envoy" pod "ff0d6342-c880-4f83-89e5-2b5e14d512ee" (UID: "ff0d6342-c880-4f83-89e5-2b5e14d512ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:50.461139 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.446088 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-token" pod "ff0d6342-c880-4f83-89e5-2b5e14d512ee" (UID: "ff0d6342-c880-4f83-89e5-2b5e14d512ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:50.461601 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.461565 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:52:50.462512 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.462271 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 03 20:52:50.462772 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.462746 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:50.467374 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.466923 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:50.467374 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.466954 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:50.468487 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.468465 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:52:50.475517 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.475495 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:52:50.478462 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.478442 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:52:50.512890 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.512743 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff0d6342-c880-4f83-89e5-2b5e14d512ee-kube-api-access-qs98q" pod "ff0d6342-c880-4f83-89e5-2b5e14d512ee" (UID: "ff0d6342-c880-4f83-89e5-2b5e14d512ee"). InnerVolumeSpecName "kube-api-access-qs98q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:50.512890 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.512836 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-podinfo" pod "ff0d6342-c880-4f83-89e5-2b5e14d512ee" (UID: "ff0d6342-c880-4f83-89e5-2b5e14d512ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:52:50.513627 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.513595 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" event={"ID":"d07d8d19-5ed5-4f0b-9084-390494a695c6","Type":"ContainerDied","Data":"a87cabaf3bc2f13cfe2175b53469a60332645f7e20deea2a13f98c611e584427"} Nov 03 20:52:50.513831 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.513815 4493 scope.go:122] "RemoveContainer" containerID="d03028ce68d694afe8f787a42a1b19eb91ab15cdc8462cba5e3cd7a729ba77a7" Nov 03 20:52:50.536448 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.536324 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" event={"ID":"c4f6d970-a5d4-4d9d-89ad-495365b7d652","Type":"ContainerDied","Data":"9fc902b4a9c4478138197c25fe2cd6bfb631bea6f130662256024252abfba1a6"} Nov 03 20:52:50.536804 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.536769 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc" Nov 03 20:52:50.627022 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.626924 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" event={"ID":"ff0d6342-c880-4f83-89e5-2b5e14d512ee","Type":"ContainerDied","Data":"80a236802b9d1b2509fd76938ca142a5dbf53defaa05215c1750818d84f3613d"} Nov 03 20:52:50.627450 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.627426 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8" Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.702234 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-podinfo\") pod \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.702316 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-credential-socket\") pod \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.702391 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-credential-socket\") pod \"a3f1f367-79f6-4068-90bc-6373d5651ab4\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.702438 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3f1f367-79f6-4068-90bc-6373d5651ab4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f1f367-79f6-4068-90bc-6373d5651ab4-istiod-ca-cert\") pod \"a3f1f367-79f6-4068-90bc-6373d5651ab4\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.702510 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-workload-socket\") pod \"d07d8d19-5ed5-4f0b-9084-390494a695c6\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.702569 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-workload-socket\") pod \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.702673 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-token\" (UniqueName: \"kubernetes.io/projected/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-token\") pod \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.702714 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-envoy\") pod \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.702797 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-workload-certs\") pod \"d07d8d19-5ed5-4f0b-9084-390494a695c6\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.702844 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-workload-socket\") pod \"a3f1f367-79f6-4068-90bc-6373d5651ab4\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.702954 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istiod-ca-cert\") pod \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.703014 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d07d8d19-5ed5-4f0b-9084-390494a695c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d07d8d19-5ed5-4f0b-9084-390494a695c6-istiod-ca-cert\") pod \"d07d8d19-5ed5-4f0b-9084-390494a695c6\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.703069 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-data\") pod \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.703112 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-data\") pod \"d07d8d19-5ed5-4f0b-9084-390494a695c6\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.703153 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-token\" (UniqueName: \"kubernetes.io/projected/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-token\") pod \"d07d8d19-5ed5-4f0b-9084-390494a695c6\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.703189 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-podinfo\") pod \"a3f1f367-79f6-4068-90bc-6373d5651ab4\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.703229 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-workload-certs\") pod \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.703289 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3f1f367-79f6-4068-90bc-6373d5651ab4-kube-api-access-fzwh9\" (UniqueName: \"kubernetes.io/projected/a3f1f367-79f6-4068-90bc-6373d5651ab4-kube-api-access-fzwh9\") pod \"a3f1f367-79f6-4068-90bc-6373d5651ab4\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.703364 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-data\") pod \"a3f1f367-79f6-4068-90bc-6373d5651ab4\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.703420 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-envoy\") pod \"a3f1f367-79f6-4068-90bc-6373d5651ab4\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.703489 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d07d8d19-5ed5-4f0b-9084-390494a695c6-kube-api-access-765j9\" (UniqueName: \"kubernetes.io/projected/d07d8d19-5ed5-4f0b-9084-390494a695c6-kube-api-access-765j9\") pod \"d07d8d19-5ed5-4f0b-9084-390494a695c6\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.703608 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-envoy\") pod \"d07d8d19-5ed5-4f0b-9084-390494a695c6\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.703664 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-workload-certs\") pod \"a3f1f367-79f6-4068-90bc-6373d5651ab4\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.703750 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4f6d970-a5d4-4d9d-89ad-495365b7d652-kube-api-access-d7hll\" (UniqueName: \"kubernetes.io/projected/c4f6d970-a5d4-4d9d-89ad-495365b7d652-kube-api-access-d7hll\") pod \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\" (UID: \"c4f6d970-a5d4-4d9d-89ad-495365b7d652\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.703812 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-credential-socket\") pod \"d07d8d19-5ed5-4f0b-9084-390494a695c6\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.703870 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-podinfo\") pod \"d07d8d19-5ed5-4f0b-9084-390494a695c6\" (UID: \"d07d8d19-5ed5-4f0b-9084-390494a695c6\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.703924 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-token\" (UniqueName: \"kubernetes.io/projected/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-token\") pod \"a3f1f367-79f6-4068-90bc-6373d5651ab4\" (UID: \"a3f1f367-79f6-4068-90bc-6373d5651ab4\") " Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.705282 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs98q\" (UniqueName: \"kubernetes.io/projected/ff0d6342-c880-4f83-89e5-2b5e14d512ee-kube-api-access-qs98q\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:50.719360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.705311 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff0d6342-c880-4f83-89e5-2b5e14d512ee-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:50.791370 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.777966 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-data" pod "d07d8d19-5ed5-4f0b-9084-390494a695c6" (UID: "d07d8d19-5ed5-4f0b-9084-390494a695c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:50.922441 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.920219 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:52:50.922441 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:50.920287 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" podUID="f62e8863-2c8c-4658-8ecf-645709ec29e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:51.066784 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.066701 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 20:52:51.221491 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.067165 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 20:52:51.221491 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.092071 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:52:51.221491 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.111486 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 20:52:51.221491 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.140555 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-workload-certs" pod "c4f6d970-a5d4-4d9d-89ad-495365b7d652" (UID: "c4f6d970-a5d4-4d9d-89ad-495365b7d652"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:51.221491 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.150741 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-data" pod "a3f1f367-79f6-4068-90bc-6373d5651ab4" (UID: "a3f1f367-79f6-4068-90bc-6373d5651ab4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:51.221491 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.213236 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-podinfo" pod "a3f1f367-79f6-4068-90bc-6373d5651ab4" (UID: "a3f1f367-79f6-4068-90bc-6373d5651ab4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:52:51.221491 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.214671 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:52:51.221491 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.218121 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" podUID="353543b3-c2cc-4563-a2a0-ddcf5b03db1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:51.221491 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.219304 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-workload-certs" pod "d07d8d19-5ed5-4f0b-9084-390494a695c6" (UID: "d07d8d19-5ed5-4f0b-9084-390494a695c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:51.221491 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.219768 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-credential-socket" pod "c4f6d970-a5d4-4d9d-89ad-495365b7d652" (UID: "c4f6d970-a5d4-4d9d-89ad-495365b7d652"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:51.221491 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.220808 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3f1f367-79f6-4068-90bc-6373d5651ab4-istiod-ca-cert" pod "a3f1f367-79f6-4068-90bc-6373d5651ab4" (UID: "a3f1f367-79f6-4068-90bc-6373d5651ab4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:51.221491 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.221312 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-workload-socket" pod "d07d8d19-5ed5-4f0b-9084-390494a695c6" (UID: "d07d8d19-5ed5-4f0b-9084-390494a695c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:51.223731 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.223478 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-credential-socket" pod "a3f1f367-79f6-4068-90bc-6373d5651ab4" (UID: "a3f1f367-79f6-4068-90bc-6373d5651ab4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:51.223731 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.223650 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-workload-socket" pod "c4f6d970-a5d4-4d9d-89ad-495365b7d652" (UID: "c4f6d970-a5d4-4d9d-89ad-495365b7d652"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:51.258115 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.257644 4493 scope.go:122] "RemoveContainer" containerID="21dae403d96d60c6909eacbc1fe3b288d73f64353f0059bd83fb18fc08193bd3" Nov 03 20:52:51.258950 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.258925 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d07d8d19-5ed5-4f0b-9084-390494a695c6-istiod-ca-cert" pod "d07d8d19-5ed5-4f0b-9084-390494a695c6" (UID: "d07d8d19-5ed5-4f0b-9084-390494a695c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:51.261366 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.261322 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-workload-certs" pod "a3f1f367-79f6-4068-90bc-6373d5651ab4" (UID: "a3f1f367-79f6-4068-90bc-6373d5651ab4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:51.267506 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.265121 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istiod-ca-cert" pod "c4f6d970-a5d4-4d9d-89ad-495365b7d652" (UID: "c4f6d970-a5d4-4d9d-89ad-495365b7d652"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:51.268294 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.267893 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-credential-socket" pod "d07d8d19-5ed5-4f0b-9084-390494a695c6" (UID: "d07d8d19-5ed5-4f0b-9084-390494a695c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:51.323397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.323362 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-workload-socket" pod "a3f1f367-79f6-4068-90bc-6373d5651ab4" (UID: "a3f1f367-79f6-4068-90bc-6373d5651ab4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:51.325376 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.324245 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 03 20:52:51.325376 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.324643 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:51.345167 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.335045 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f1f367-79f6-4068-90bc-6373d5651ab4-kube-api-access-fzwh9" pod "a3f1f367-79f6-4068-90bc-6373d5651ab4" (UID: "a3f1f367-79f6-4068-90bc-6373d5651ab4"). InnerVolumeSpecName "kube-api-access-fzwh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:51.367395 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.348530 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-token" pod "c4f6d970-a5d4-4d9d-89ad-495365b7d652" (UID: "c4f6d970-a5d4-4d9d-89ad-495365b7d652"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:51.367395 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.348643 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d07d8d19-5ed5-4f0b-9084-390494a695c6-kube-api-access-765j9" pod "d07d8d19-5ed5-4f0b-9084-390494a695c6" (UID: "d07d8d19-5ed5-4f0b-9084-390494a695c6"). InnerVolumeSpecName "kube-api-access-765j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:51.380964 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.380920 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f6d970-a5d4-4d9d-89ad-495365b7d652-kube-api-access-d7hll" pod "c4f6d970-a5d4-4d9d-89ad-495365b7d652" (UID: "c4f6d970-a5d4-4d9d-89ad-495365b7d652"). InnerVolumeSpecName "kube-api-access-d7hll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:51.466907 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.449975 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-token" pod "a3f1f367-79f6-4068-90bc-6373d5651ab4" (UID: "a3f1f367-79f6-4068-90bc-6373d5651ab4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:51.469558 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.469403 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-data" pod "c4f6d970-a5d4-4d9d-89ad-495365b7d652" (UID: "c4f6d970-a5d4-4d9d-89ad-495365b7d652"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:51.488076 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.487992 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 03 20:52:51.489865 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.489802 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:51.490938 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.490913 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:52:51.492052 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.491923 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-token" pod "d07d8d19-5ed5-4f0b-9084-390494a695c6" (UID: "d07d8d19-5ed5-4f0b-9084-390494a695c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:51.497947 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.495070 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.532402 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.532326 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-envoy" pod "d07d8d19-5ed5-4f0b-9084-390494a695c6" (UID: "d07d8d19-5ed5-4f0b-9084-390494a695c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:51.533242 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.532659 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-envoy" pod "c4f6d970-a5d4-4d9d-89ad-495365b7d652" (UID: "c4f6d970-a5d4-4d9d-89ad-495365b7d652"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:51.548293 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.547766 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-envoy" pod "a3f1f367-79f6-4068-90bc-6373d5651ab4" (UID: "a3f1f367-79f6-4068-90bc-6373d5651ab4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:51.573389 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.573364 4493 scope.go:122] "RemoveContainer" containerID="f1dc2286ce1c2afe2160af159cad5e1cc85a5ec67d65973346ac0e9aeff3cf5c" Nov 03 20:52:51.580593 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.580463 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-podinfo" pod "d07d8d19-5ed5-4f0b-9084-390494a695c6" (UID: "d07d8d19-5ed5-4f0b-9084-390494a695c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:52:51.609543 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.608358 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d07d8d19-5ed5-4f0b-9084-390494a695c6-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.609711 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.609691 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.609828 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.609815 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.609923 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.609904 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.610023 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.610011 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.610117 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.610105 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzwh9\" (UniqueName: \"kubernetes.io/projected/a3f1f367-79f6-4068-90bc-6373d5651ab4-kube-api-access-fzwh9\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.610193 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.610182 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.610280 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.610268 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.610403 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.610389 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-765j9\" (UniqueName: \"kubernetes.io/projected/d07d8d19-5ed5-4f0b-9084-390494a695c6-kube-api-access-765j9\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.621065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.620630 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.621065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.620671 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.621065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.620689 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7hll\" (UniqueName: \"kubernetes.io/projected/c4f6d970-a5d4-4d9d-89ad-495365b7d652-kube-api-access-d7hll\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.621065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.620719 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.621065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.620734 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d07d8d19-5ed5-4f0b-9084-390494a695c6-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.621065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.620750 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3f1f367-79f6-4068-90bc-6373d5651ab4-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.621065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.620764 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.621065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.620786 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.621065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.620801 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f1f367-79f6-4068-90bc-6373d5651ab4-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.621065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.620815 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.621065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.620844 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.621065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.620870 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.621065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.620886 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.621065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.620901 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d07d8d19-5ed5-4f0b-9084-390494a695c6-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.621065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.620920 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f1f367-79f6-4068-90bc-6373d5651ab4-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.621065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.620936 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.621065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.614234 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:52:51.636483 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.636460 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-podinfo" pod "c4f6d970-a5d4-4d9d-89ad-495365b7d652" (UID: "c4f6d970-a5d4-4d9d-89ad-495365b7d652"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:52:51.651799 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.651773 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l" Nov 03 20:52:51.652510 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.652490 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4" Nov 03 20:52:51.656467 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.656427 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:51.668728 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.668707 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8"] Nov 03 20:52:51.668852 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.668833 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-85pk8"] Nov 03 20:52:51.674686 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.670972 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:52:51.689130 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.689100 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:52:51.689746 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.689722 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:52:51.689883 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.689775 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:52:51.689883 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.689810 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:52:51.700095 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.700060 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:52:51.700397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.700369 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:52:51.702177 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.701132 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:52:51.727033 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:51.723388 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4f6d970-a5d4-4d9d-89ad-495365b7d652-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.766010 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:52:51.765976 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:52:52.562972 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:52.562891 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff0d6342-c880-4f83-89e5-2b5e14d512ee" path="/var/lib/kubelet/pods/ff0d6342-c880-4f83-89e5-2b5e14d512ee/volumes" Nov 03 20:52:52.619371 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:52.619319 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l"] Nov 03 20:52:52.624295 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:52.624257 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9pl8l"] Nov 03 20:52:52.719267 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:52.719239 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4"] Nov 03 20:52:52.723261 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:52.723236 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-dbbg4"] Nov 03 20:52:52.948600 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:52.948508 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc"] Nov 03 20:52:52.953170 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:52.953141 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-5qqpc"] Nov 03 20:52:54.564509 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:54.564474 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3f1f367-79f6-4068-90bc-6373d5651ab4" path="/var/lib/kubelet/pods/a3f1f367-79f6-4068-90bc-6373d5651ab4/volumes" Nov 03 20:52:54.565097 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:54.564961 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4f6d970-a5d4-4d9d-89ad-495365b7d652" path="/var/lib/kubelet/pods/c4f6d970-a5d4-4d9d-89ad-495365b7d652/volumes" Nov 03 20:52:54.565397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:54.565382 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d07d8d19-5ed5-4f0b-9084-390494a695c6" path="/var/lib/kubelet/pods/d07d8d19-5ed5-4f0b-9084-390494a695c6/volumes" Nov 03 20:52:55.243207 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.243162 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq"] Nov 03 20:52:55.243888 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.243866 4493 cpu_manager.go:414] "containerMap: removing container" podUID="d07d8d19-5ed5-4f0b-9084-390494a695c6" containerName="istio-proxy" Nov 03 20:52:55.243888 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.243888 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d07d8d19-5ed5-4f0b-9084-390494a695c6" containerName="istio-proxy" Nov 03 20:52:55.244075 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.243926 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ff0d6342-c880-4f83-89e5-2b5e14d512ee" containerName="istio-proxy" Nov 03 20:52:55.244075 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.243936 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff0d6342-c880-4f83-89e5-2b5e14d512ee" containerName="istio-proxy" Nov 03 20:52:55.244075 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.243958 4493 cpu_manager.go:414] "containerMap: removing container" podUID="c4f6d970-a5d4-4d9d-89ad-495365b7d652" containerName="istio-proxy" Nov 03 20:52:55.244075 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.243966 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4f6d970-a5d4-4d9d-89ad-495365b7d652" containerName="istio-proxy" Nov 03 20:52:55.244075 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.243979 4493 cpu_manager.go:414] "containerMap: removing container" podUID="a3f1f367-79f6-4068-90bc-6373d5651ab4" containerName="istio-proxy" Nov 03 20:52:55.244075 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.243985 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3f1f367-79f6-4068-90bc-6373d5651ab4" containerName="istio-proxy" Nov 03 20:52:55.244665 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.244124 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4f6d970-a5d4-4d9d-89ad-495365b7d652" containerName="istio-proxy" Nov 03 20:52:55.244665 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.244136 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d07d8d19-5ed5-4f0b-9084-390494a695c6" containerName="istio-proxy" Nov 03 20:52:55.244665 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.244148 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3f1f367-79f6-4068-90bc-6373d5651ab4" containerName="istio-proxy" Nov 03 20:52:55.244665 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.244167 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff0d6342-c880-4f83-89e5-2b5e14d512ee" containerName="istio-proxy" Nov 03 20:52:55.250547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.250524 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.290356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.277416 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.290356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.277526 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.290356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.277569 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.290356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.277695 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.290356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.277797 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30161e35-b355-4b5f-8746-92522021c10a-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.290356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.277913 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30161e35-b355-4b5f-8746-92522021c10a-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.290356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.277945 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30161e35-b355-4b5f-8746-92522021c10a-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.290356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.278021 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.290356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.278050 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvn9h\" (UniqueName: \"kubernetes.io/projected/30161e35-b355-4b5f-8746-92522021c10a-kube-api-access-cvn9h\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.316908 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.316846 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq"] Nov 03 20:52:55.394440 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.379287 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4"] Nov 03 20:52:55.394440 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.381371 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.394440 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.381486 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30161e35-b355-4b5f-8746-92522021c10a-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.394440 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.381570 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30161e35-b355-4b5f-8746-92522021c10a-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.394440 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.381607 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30161e35-b355-4b5f-8746-92522021c10a-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.394440 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.381682 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.394440 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.381713 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvn9h\" (UniqueName: \"kubernetes.io/projected/30161e35-b355-4b5f-8746-92522021c10a-kube-api-access-cvn9h\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.394440 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.381760 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.394440 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.381847 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.394440 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.381886 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.394440 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.382408 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.403395 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.403028 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30161e35-b355-4b5f-8746-92522021c10a-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.404902 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.404876 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.405308 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.405274 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.405849 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.405755 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.406062 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.406042 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.406317 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.406299 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.406687 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.406668 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30161e35-b355-4b5f-8746-92522021c10a-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.468139 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.468112 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-43693d42-openshift-default-dockercfg-lrmf4\"" Nov 03 20:52:55.471250 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.471217 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30161e35-b355-4b5f-8746-92522021c10a-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.471466 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.471313 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4"] Nov 03 20:52:55.475606 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.475583 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvn9h\" (UniqueName: \"kubernetes.io/projected/30161e35-b355-4b5f-8746-92522021c10a-kube-api-access-cvn9h\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.483012 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.482986 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxmjs\" (UniqueName: \"kubernetes.io/projected/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-kube-api-access-lxmjs\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.483115 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.483023 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-envoy\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.483298 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.483270 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-workload-certs\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.483524 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.483502 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-workload-socket\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.484791 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.483543 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-token\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.484791 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.483717 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-data\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.484791 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.483774 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-podinfo\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.484791 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.483803 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istiod-ca-cert\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.484791 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.483840 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-credential-socket\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.570288 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.570248 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:55.585101 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.584987 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-workload-socket\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.585101 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.585042 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-token\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.585308 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.585147 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-data\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.585308 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.585198 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-podinfo\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.585308 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.585224 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istiod-ca-cert\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.585308 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.585278 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-credential-socket\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.585629 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.585318 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxmjs\" (UniqueName: \"kubernetes.io/projected/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-kube-api-access-lxmjs\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.585629 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.585383 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-envoy\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.585629 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.585521 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-workload-certs\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.586877 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.586851 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istiod-ca-cert\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.587112 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.587087 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-credential-socket\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.587662 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.587625 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-workload-certs\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.588221 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.587890 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-workload-socket\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.588377 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.588257 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-data\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.589305 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.589278 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-podinfo\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.590859 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.590833 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-envoy\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.651255 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.651223 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxmjs\" (UniqueName: \"kubernetes.io/projected/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-kube-api-access-lxmjs\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.651776 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.651753 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-token\") pod \"e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:55.757190 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:55.757146 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:56.008427 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:56.008385 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq"] Nov 03 20:52:56.013985 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:52:56.013953 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30161e35_b355_4b5f_8746_92522021c10a.slice/crio-0c1fe68b8ec87cd3d9c082370df17d902f41fd58c88d897184ea0b0ff979b560 WatchSource:0}: Error finding container 0c1fe68b8ec87cd3d9c082370df17d902f41fd58c88d897184ea0b0ff979b560: Status 404 returned error can't find the container with id 0c1fe68b8ec87cd3d9c082370df17d902f41fd58c88d897184ea0b0ff979b560 Nov 03 20:52:56.016841 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:56.016389 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:52:56.017100 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:56.017076 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:52:56.017295 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:56.017269 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:52:56.153408 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:56.153370 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4"] Nov 03 20:52:56.167675 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:56.167624 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:52:56.167766 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:56.167732 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:52:56.167804 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:56.167780 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:52:56.719807 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:56.719768 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" event={"ID":"30161e35-b355-4b5f-8746-92522021c10a","Type":"ContainerStarted","Data":"778efe557a3ce49aa96899027d7cbadb336384edabe7a881d087471f39c2387c"} Nov 03 20:52:56.720293 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:56.719832 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" event={"ID":"30161e35-b355-4b5f-8746-92522021c10a","Type":"ContainerStarted","Data":"0c1fe68b8ec87cd3d9c082370df17d902f41fd58c88d897184ea0b0ff979b560"} Nov 03 20:52:56.721676 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:56.721652 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" event={"ID":"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1","Type":"ContainerStarted","Data":"0b55a24f7062046592a97b6e3d19b652f78dc88847b1bb4386d02dcf6eee2c24"} Nov 03 20:52:56.721814 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:56.721702 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" event={"ID":"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1","Type":"ContainerStarted","Data":"3fd622843c423bd469dc3a9bb773a311ba0a416a5e7daf128df674603cd51029"} Nov 03 20:52:56.758155 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:56.758127 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:56.831164 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:56.831105 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" podStartSLOduration=1.831092268 podStartE2EDuration="1.831092268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:56.829398565 +0000 UTC m=+1448.387389065" watchObservedRunningTime="2026-11-03 20:52:56.831092268 +0000 UTC m=+1448.389082767" Nov 03 20:52:56.941570 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:56.941519 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" podStartSLOduration=1.941503547 podStartE2EDuration="1.941503547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:56.939650864 +0000 UTC m=+1448.497641364" watchObservedRunningTime="2026-11-03 20:52:56.941503547 +0000 UTC m=+1448.499494046" Nov 03 20:52:57.571041 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:57.571007 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:52:57.572312 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:57.572288 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:52:57.572450 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:57.572347 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" podUID="30161e35-b355-4b5f-8746-92522021c10a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 20:52:57.764944 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:57.764909 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:58.570630 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:58.570600 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:52:58.570798 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:58.570649 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" podUID="30161e35-b355-4b5f-8746-92522021c10a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 20:52:58.737781 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:58.737749 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:58.738855 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:58.738822 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:52:59.571139 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:59.571107 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:52:59.571561 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:52:59.571168 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" podUID="30161e35-b355-4b5f-8746-92522021c10a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 20:53:00.570626 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:00.570597 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:53:00.570826 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:00.570662 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" podUID="30161e35-b355-4b5f-8746-92522021c10a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 20:53:01.571025 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:01.570988 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:53:01.571565 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:01.571064 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" podUID="30161e35-b355-4b5f-8746-92522021c10a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 20:53:02.570660 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:02.570624 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:53:02.570857 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:02.570688 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" podUID="30161e35-b355-4b5f-8746-92522021c10a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 20:53:03.575413 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:03.575379 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:53:03.575934 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:03.575773 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:53:03.576581 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:03.576560 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 20:53:04.527809 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.527779 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7"] Nov 03 20:53:04.533846 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.533823 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.595444 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.594520 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7"] Nov 03 20:53:04.606364 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.606298 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-1a55d7db-openshift-default-dockercfg-bc66g\"" Nov 03 20:53:04.677835 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.677799 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-credential-socket\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.677999 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.677919 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-data\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.677999 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.677951 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w78l2\" (UniqueName: \"kubernetes.io/projected/1f96068d-fb57-4a30-876a-e3158d210c3c-kube-api-access-w78l2\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.678071 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.678036 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-workload-certs\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.678118 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.678103 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-workload-socket\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.678244 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.678218 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f96068d-fb57-4a30-876a-e3158d210c3c-istiod-ca-cert\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.678407 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.678383 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-envoy\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.678529 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.678510 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-podinfo\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.678640 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.678620 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-token\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.780494 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.780407 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f96068d-fb57-4a30-876a-e3158d210c3c-istiod-ca-cert\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.780494 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.780452 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-envoy\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.780734 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.780502 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-podinfo\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.780734 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.780650 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-token\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.780840 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.780740 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-credential-socket\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.780902 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.780835 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-data\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.780902 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.780881 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w78l2\" (UniqueName: \"kubernetes.io/projected/1f96068d-fb57-4a30-876a-e3158d210c3c-kube-api-access-w78l2\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.781034 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.780965 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-workload-certs\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.781095 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.781058 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-workload-socket\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.781261 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.781211 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-data\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.781450 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.781274 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f96068d-fb57-4a30-876a-e3158d210c3c-istiod-ca-cert\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.781534 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.781461 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-workload-socket\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.781605 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.781565 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-credential-socket\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.781605 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.781580 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-workload-certs\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.783719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.783692 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-envoy\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.783912 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.783890 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-podinfo\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.837764 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.837732 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w78l2\" (UniqueName: \"kubernetes.io/projected/1f96068d-fb57-4a30-876a-e3158d210c3c-kube-api-access-w78l2\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.837957 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.837934 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-token\") pod \"e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:04.851020 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:04.850994 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:05.316675 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:05.316643 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7"] Nov 03 20:53:05.321320 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:53:05.321292 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f96068d_fb57_4a30_876a_e3158d210c3c.slice/crio-9fdb3c66f74e1f3ccc224270f75576abca18d9772a59b55d19b7abf15a049822 WatchSource:0}: Error finding container 9fdb3c66f74e1f3ccc224270f75576abca18d9772a59b55d19b7abf15a049822: Status 404 returned error can't find the container with id 9fdb3c66f74e1f3ccc224270f75576abca18d9772a59b55d19b7abf15a049822 Nov 03 20:53:05.323853 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:05.323819 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:53:05.323945 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:05.323885 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:53:05.323945 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:05.323916 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:53:05.807899 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:05.807868 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" event={"ID":"1f96068d-fb57-4a30-876a-e3158d210c3c","Type":"ContainerStarted","Data":"eb5d8bdef9431fa1341ba5ed49b46cbf67a0875fde27af1822fd63ba0b5c3702"} Nov 03 20:53:05.808271 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:05.807923 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" event={"ID":"1f96068d-fb57-4a30-876a-e3158d210c3c","Type":"ContainerStarted","Data":"9fdb3c66f74e1f3ccc224270f75576abca18d9772a59b55d19b7abf15a049822"} Nov 03 20:53:05.851305 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:05.851275 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:05.924192 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:05.924133 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" podStartSLOduration=1.924115954 podStartE2EDuration="1.924115954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:05.923545999 +0000 UTC m=+1457.481536499" watchObservedRunningTime="2026-11-03 20:53:05.924115954 +0000 UTC m=+1457.482106454" Nov 03 20:53:06.857600 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:06.857571 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:07.821387 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:07.821362 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:07.822456 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:07.822432 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:53:16.326700 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:16.326666 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w"] Nov 03 20:53:16.331088 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:16.331069 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w" Nov 03 20:53:16.384535 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:16.384507 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w"] Nov 03 20:53:16.427503 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:16.427471 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-43693d42\"" Nov 03 20:53:16.427713 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:16.427670 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-43693d42-dockercfg-dp6gm\"" Nov 03 20:53:16.509952 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:16.509911 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-43693d42-844854c99c-wkh7b"] Nov 03 20:53:16.512487 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:16.512452 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj8l5\" (UniqueName: \"kubernetes.io/projected/2b783dd2-815d-40be-b790-4e303c07e82c-kube-api-access-sj8l5\") pod \"payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w\" (UID: \"2b783dd2-815d-40be-b790-4e303c07e82c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w" Nov 03 20:53:16.512696 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:16.512671 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b783dd2-815d-40be-b790-4e303c07e82c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w\" (UID: \"2b783dd2-815d-40be-b790-4e303c07e82c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w" Nov 03 20:53:16.514637 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:16.514588 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-43693d42-844854c99c-wkh7b" Nov 03 20:53:16.535757 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:16.535704 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-43693d42-844854c99c-wkh7b"] Nov 03 20:53:16.614540 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:16.614444 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj8l5\" (UniqueName: \"kubernetes.io/projected/2b783dd2-815d-40be-b790-4e303c07e82c-kube-api-access-sj8l5\") pod \"payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w\" (UID: \"2b783dd2-815d-40be-b790-4e303c07e82c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w" Nov 03 20:53:16.614540 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:16.614523 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpsrg\" (UniqueName: \"kubernetes.io/projected/b0fec100-98d7-4570-860e-0d89ab70f264-kube-api-access-vpsrg\") pod \"payload-processing-e2e-ait-43693d42-844854c99c-wkh7b\" (UID: \"b0fec100-98d7-4570-860e-0d89ab70f264\") " pod="openshift-ingress/payload-processing-e2e-ait-43693d42-844854c99c-wkh7b" Nov 03 20:53:16.614789 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:16.614566 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0fec100-98d7-4570-860e-0d89ab70f264-plugins-config-volume\") pod \"payload-processing-e2e-ait-43693d42-844854c99c-wkh7b\" (UID: \"b0fec100-98d7-4570-860e-0d89ab70f264\") " pod="openshift-ingress/payload-processing-e2e-ait-43693d42-844854c99c-wkh7b" Nov 03 20:53:16.614789 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:16.614733 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b783dd2-815d-40be-b790-4e303c07e82c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w\" (UID: \"2b783dd2-815d-40be-b790-4e303c07e82c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w" Nov 03 20:53:16.615775 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:16.615746 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b783dd2-815d-40be-b790-4e303c07e82c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w\" (UID: \"2b783dd2-815d-40be-b790-4e303c07e82c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w" Nov 03 20:53:16.671932 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:16.671904 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj8l5\" (UniqueName: \"kubernetes.io/projected/2b783dd2-815d-40be-b790-4e303c07e82c-kube-api-access-sj8l5\") pod \"payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w\" (UID: \"2b783dd2-815d-40be-b790-4e303c07e82c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w" Nov 03 20:53:16.717191 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:16.717143 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpsrg\" (UniqueName: \"kubernetes.io/projected/b0fec100-98d7-4570-860e-0d89ab70f264-kube-api-access-vpsrg\") pod \"payload-processing-e2e-ait-43693d42-844854c99c-wkh7b\" (UID: \"b0fec100-98d7-4570-860e-0d89ab70f264\") " pod="openshift-ingress/payload-processing-e2e-ait-43693d42-844854c99c-wkh7b" Nov 03 20:53:16.717434 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:16.717263 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0fec100-98d7-4570-860e-0d89ab70f264-plugins-config-volume\") pod \"payload-processing-e2e-ait-43693d42-844854c99c-wkh7b\" (UID: \"b0fec100-98d7-4570-860e-0d89ab70f264\") " pod="openshift-ingress/payload-processing-e2e-ait-43693d42-844854c99c-wkh7b" Nov 03 20:53:16.718297 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:16.718273 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0fec100-98d7-4570-860e-0d89ab70f264-plugins-config-volume\") pod \"payload-processing-e2e-ait-43693d42-844854c99c-wkh7b\" (UID: \"b0fec100-98d7-4570-860e-0d89ab70f264\") " pod="openshift-ingress/payload-processing-e2e-ait-43693d42-844854c99c-wkh7b" Nov 03 20:53:16.774766 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:16.774732 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpsrg\" (UniqueName: \"kubernetes.io/projected/b0fec100-98d7-4570-860e-0d89ab70f264-kube-api-access-vpsrg\") pod \"payload-processing-e2e-ait-43693d42-844854c99c-wkh7b\" (UID: \"b0fec100-98d7-4570-860e-0d89ab70f264\") " pod="openshift-ingress/payload-processing-e2e-ait-43693d42-844854c99c-wkh7b" Nov 03 20:53:16.839789 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:16.839757 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-43693d42-844854c99c-wkh7b" Nov 03 20:53:16.948989 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:16.946190 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w" Nov 03 20:53:17.295933 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:17.295899 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-43693d42-844854c99c-wkh7b"] Nov 03 20:53:17.300578 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:53:17.300542 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0fec100_98d7_4570_860e_0d89ab70f264.slice/crio-31c120d7517360437e633c8a97316eb3dc8fa8f9de7471bac30bac8d6d1d0225 WatchSource:0}: Error finding container 31c120d7517360437e633c8a97316eb3dc8fa8f9de7471bac30bac8d6d1d0225: Status 404 returned error can't find the container with id 31c120d7517360437e633c8a97316eb3dc8fa8f9de7471bac30bac8d6d1d0225 Nov 03 20:53:17.302561 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:17.302543 4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:53:17.449699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:17.449670 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w"] Nov 03 20:53:17.453950 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:53:17.453926 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b783dd2_815d_40be_b790_4e303c07e82c.slice/crio-bac30682d1e9d25560c450370c8b9f9c1d39f65ab739385970e29924350f57b1 WatchSource:0}: Error finding container bac30682d1e9d25560c450370c8b9f9c1d39f65ab739385970e29924350f57b1: Status 404 returned error can't find the container with id bac30682d1e9d25560c450370c8b9f9c1d39f65ab739385970e29924350f57b1 Nov 03 20:53:17.928606 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:17.928571 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt"] Nov 03 20:53:17.934389 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:17.934364 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:17.964365 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:17.963748 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-workload-socket\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:17.964365 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:17.963851 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4c678e-8285-4d16-984d-39fa208f1ae8-istiod-ca-cert\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:17.964365 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:17.963887 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpvjm\" (UniqueName: \"kubernetes.io/projected/9f4c678e-8285-4d16-984d-39fa208f1ae8-kube-api-access-lpvjm\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:17.964365 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:17.963961 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-data\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:17.964365 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:17.964000 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-envoy\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:17.964365 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:17.964035 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-workload-certs\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:17.964365 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:17.964063 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-token\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:17.964365 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:17.964158 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-podinfo\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:17.964365 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:17.964209 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-credential-socket\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:18.050393 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.022240 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-43693d42-844854c99c-wkh7b" event={"ID":"b0fec100-98d7-4570-860e-0d89ab70f264","Type":"ContainerStarted","Data":"31c120d7517360437e633c8a97316eb3dc8fa8f9de7471bac30bac8d6d1d0225"} Nov 03 20:53:18.050393 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.023917 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w" event={"ID":"2b783dd2-815d-40be-b790-4e303c07e82c","Type":"ContainerStarted","Data":"bac30682d1e9d25560c450370c8b9f9c1d39f65ab739385970e29924350f57b1"} Nov 03 20:53:18.050393 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.024722 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt"] Nov 03 20:53:18.050393 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.037650 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e3e5cd62-openshift-default-dockercfg-t42jm\"" Nov 03 20:53:18.092232 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.091132 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-credential-socket\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:18.092232 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.091393 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-workload-socket\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:18.092232 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.091681 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4c678e-8285-4d16-984d-39fa208f1ae8-istiod-ca-cert\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:18.092232 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.091746 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpvjm\" (UniqueName: \"kubernetes.io/projected/9f4c678e-8285-4d16-984d-39fa208f1ae8-kube-api-access-lpvjm\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:18.092232 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.091843 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-workload-socket\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:18.092232 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.091847 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-data\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:18.092232 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.091921 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-envoy\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:18.092232 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.091959 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-workload-certs\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:18.092232 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.091989 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-token\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:18.092232 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.092066 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-podinfo\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:18.092911 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.092327 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-credential-socket\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:18.092911 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.092820 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-data\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:18.093080 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.093056 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-workload-certs\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:18.094030 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.094004 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4c678e-8285-4d16-984d-39fa208f1ae8-istiod-ca-cert\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:18.098232 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.097312 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-envoy\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:18.098232 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.097926 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-podinfo\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:18.147355 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.147216 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-token\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:18.149629 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.148738 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpvjm\" (UniqueName: \"kubernetes.io/projected/9f4c678e-8285-4d16-984d-39fa208f1ae8-kube-api-access-lpvjm\") pod \"e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:18.330222 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.330188 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:18.802207 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.801843 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt"] Nov 03 20:53:18.804889 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:53:18.804843 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4c678e_8285_4d16_984d_39fa208f1ae8.slice/crio-1809a30a603767ce6192d5b96d5ab0eeaac145c9615a899a1db6c04277fe092e WatchSource:0}: Error finding container 1809a30a603767ce6192d5b96d5ab0eeaac145c9615a899a1db6c04277fe092e: Status 404 returned error can't find the container with id 1809a30a603767ce6192d5b96d5ab0eeaac145c9615a899a1db6c04277fe092e Nov 03 20:53:18.808314 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.808277 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:53:18.808432 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.808384 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:53:18.808432 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:18.808427 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:53:19.038575 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:19.038464 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" event={"ID":"9f4c678e-8285-4d16-984d-39fa208f1ae8","Type":"ContainerStarted","Data":"1a4e7de59a82a80ef3ef355cd595031cf005610156f6488827b292e41cc035ee"} Nov 03 20:53:19.038575 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:19.038536 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" event={"ID":"9f4c678e-8285-4d16-984d-39fa208f1ae8","Type":"ContainerStarted","Data":"1809a30a603767ce6192d5b96d5ab0eeaac145c9615a899a1db6c04277fe092e"} Nov 03 20:53:19.152013 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:19.151877 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" podStartSLOduration=2.151858016 podStartE2EDuration="2.151858016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:19.149015983 +0000 UTC m=+1470.707006505" watchObservedRunningTime="2026-11-03 20:53:19.151858016 +0000 UTC m=+1470.709848516" Nov 03 20:53:19.330699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:19.330665 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:19.332240 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:19.332207 4493 patch_prober.go:29] interesting pod/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 20:53:19.332420 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:19.332269 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" podUID="9f4c678e-8285-4d16-984d-39fa208f1ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 20:53:20.398356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:20.397675 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:21.113360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:21.110247 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:21.126362 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:21.122572 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:53:21.495443 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:21.494433 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 20:53:21.521358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:21.507469 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 20:53:21.664727 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:21.664681 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:21.665122 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:21.664967 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:21.665377 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:21.665017 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:21.665540 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:21.665039 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 20:53:22.180373 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:22.165640 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded" Nov 03 20:53:22.190396 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:22.187510 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:22.202227 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:22.201667 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:53:22.202989 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:53:22.202911 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:53:22.309917 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:22.309875 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:22.313692 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:22.310913 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:53:22.499647 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:22.499524 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:22.501191 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:22.501151 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:53:22.521989 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:22.521831 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:22.522957 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:22.522916 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:53:22.681477 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:22.681431 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:22.690371 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:22.681737 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:53:22.690371 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:22.681773 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:22.910365 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:22.909612 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:53:22.910365 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:22.909704 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:53:22.910365 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:22.909766 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:53:23.211412 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:23.200768 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:23.546999 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:23.219679 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:53:23.546999 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:23.300151 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:53:23.546999 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:23.501408 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:53:23.546999 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:23.501457 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:53:23.546999 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:23.317736 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 03 20:53:23.546999 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:23.502029 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:53:23.546999 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:23.317806 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:53:23.546999 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:23.503434 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:23.546999 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:23.503749 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:53:23.581723 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:23.560232 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:53:23.694012 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:23.693970 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 03 20:53:23.694962 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:23.694932 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:53:23.962321 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:23.962243 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:53:24.011727 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:24.011662 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:53:24.052851 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:24.052827 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:53:24.076212 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:24.076175 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:53:24.100680 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:24.100654 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:53:24.160826 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:24.160175 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:53:24.185048 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:24.185021 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:53:24.546543 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:24.544108 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:53:24.546543 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:24.544305 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" podUID="f1a56925-1c64-4165-abf5-3c9cc1cf22f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:24.754633 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:24.754599 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn"] Nov 03 20:53:24.776942 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:24.776904 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn"] Nov 03 20:53:24.777108 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:24.776969 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:24.929970 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:24.929158 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:24.929970 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:24.929302 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:24.929970 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:24.929352 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:24.929970 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:24.929413 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:24.929970 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:24.929453 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rxpv\" (UniqueName: \"kubernetes.io/projected/0df130a3-4a40-4837-8211-cfc0ed20d0c2-kube-api-access-8rxpv\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:24.929970 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:24.929514 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:24.929970 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:24.929615 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:24.929970 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:24.929639 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:24.929970 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:24.929700 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:25.030803 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:25.030766 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:25.030995 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:25.030816 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:25.030995 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:25.030881 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:25.031111 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:25.031008 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:25.031167 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:25.031140 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:25.031232 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:25.031171 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:25.031232 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:25.031207 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:25.031232 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:25.031223 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:25.031407 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:25.031269 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rxpv\" (UniqueName: \"kubernetes.io/projected/0df130a3-4a40-4837-8211-cfc0ed20d0c2-kube-api-access-8rxpv\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:25.031407 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:25.031380 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:25.031949 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:25.031530 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:25.031949 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:25.031802 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:25.032323 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:25.032296 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:25.032323 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:25.032315 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:25.034732 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:25.034687 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:25.034840 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:25.034813 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:25.090938 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:25.090877 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:25.091689 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:25.091651 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rxpv\" (UniqueName: \"kubernetes.io/projected/0df130a3-4a40-4837-8211-cfc0ed20d0c2-kube-api-access-8rxpv\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:25.097068 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:25.097042 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:25.548110 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:25.548079 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn"] Nov 03 20:53:25.552576 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:53:25.552546 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df130a3_4a40_4837_8211_cfc0ed20d0c2.slice/crio-fcbd1a2f594c0716edf000a49e66b4ecb524ef7ece30029c96c2b9c37ddd253d WatchSource:0}: Error finding container fcbd1a2f594c0716edf000a49e66b4ecb524ef7ece30029c96c2b9c37ddd253d: Status 404 returned error can't find the container with id fcbd1a2f594c0716edf000a49e66b4ecb524ef7ece30029c96c2b9c37ddd253d Nov 03 20:53:25.555663 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:25.555624 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:53:25.555759 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:25.555701 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:53:25.555759 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:25.555733 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:53:26.460110 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:26.460076 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w" event={"ID":"2b783dd2-815d-40be-b790-4e303c07e82c","Type":"ContainerStarted","Data":"09243faedf52d27d133a6cc95cc6f34c38c9ca3652923301215762ec97308cd1"} Nov 03 20:53:26.460598 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:26.460302 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w" Nov 03 20:53:26.461889 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:26.461860 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-43693d42-844854c99c-wkh7b" event={"ID":"b0fec100-98d7-4570-860e-0d89ab70f264","Type":"ContainerStarted","Data":"de0f4f7e19be58d0404afa37965a6bfd33e11be9ad5dc4afaac5ee056f7e98c7"} Nov 03 20:53:26.462016 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:26.461922 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-43693d42-844854c99c-wkh7b" Nov 03 20:53:26.463525 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:26.463502 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" event={"ID":"0df130a3-4a40-4837-8211-cfc0ed20d0c2","Type":"ContainerStarted","Data":"61104a1295d86f46df6e7905192f0a215308438f5977ac4efff64b6ace6de397"} Nov 03 20:53:26.463653 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:26.463547 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" event={"ID":"0df130a3-4a40-4837-8211-cfc0ed20d0c2","Type":"ContainerStarted","Data":"fcbd1a2f594c0716edf000a49e66b4ecb524ef7ece30029c96c2b9c37ddd253d"} Nov 03 20:53:26.680645 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:26.680590 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w" podStartSLOduration=2.300996216 podStartE2EDuration="10.680575075s" totalImagesPullingTime="8.379578859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:26.571709481 +0000 UTC m=+1478.129699980" watchObservedRunningTime="2026-11-03 20:53:26.680575075 +0000 UTC m=+1478.238565626" Nov 03 20:53:26.681852 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:26.681802 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-43693d42-844854c99c-wkh7b" podStartSLOduration=2.250303321 podStartE2EDuration="10.681791985s" totalImagesPullingTime="8.431488664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:26.67830916 +0000 UTC m=+1478.236299665" watchObservedRunningTime="2026-11-03 20:53:26.681791985 +0000 UTC m=+1478.239782484" Nov 03 20:53:26.811083 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:26.811023 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" podStartSLOduration=3.811001661 podStartE2EDuration="3.811001661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:26.80888575 +0000 UTC m=+1478.366876250" watchObservedRunningTime="2026-11-03 20:53:26.811001661 +0000 UTC m=+1478.368992162" Nov 03 20:53:27.031122 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:27.031089 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw"] Nov 03 20:53:27.039461 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:27.038499 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw" Nov 03 20:53:27.098930 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:27.098843 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:27.099897 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:27.099867 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-1a55d7db-dockercfg-f2bfd\"" Nov 03 20:53:27.100748 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:27.100724 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:53:27.100945 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:27.100914 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" podUID="0df130a3-4a40-4837-8211-cfc0ed20d0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 20:53:27.101453 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:27.101433 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-1a55d7db\"" Nov 03 20:53:27.126591 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:27.126555 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw"] Nov 03 20:53:27.174712 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:27.174675 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd8hr\" (UniqueName: \"kubernetes.io/projected/140b531b-f63d-42aa-bbc6-62059f6b7aba-kube-api-access-xd8hr\") pod \"payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw\" (UID: \"140b531b-f63d-42aa-bbc6-62059f6b7aba\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw" Nov 03 20:53:27.174911 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:27.174814 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/140b531b-f63d-42aa-bbc6-62059f6b7aba-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw\" (UID: \"140b531b-f63d-42aa-bbc6-62059f6b7aba\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw" Nov 03 20:53:27.368107 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:27.366547 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/140b531b-f63d-42aa-bbc6-62059f6b7aba-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw\" (UID: \"140b531b-f63d-42aa-bbc6-62059f6b7aba\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw" Nov 03 20:53:27.368107 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:27.366944 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd8hr\" (UniqueName: \"kubernetes.io/projected/140b531b-f63d-42aa-bbc6-62059f6b7aba-kube-api-access-xd8hr\") pod \"payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw\" (UID: \"140b531b-f63d-42aa-bbc6-62059f6b7aba\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw" Nov 03 20:53:27.368107 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:27.367998 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/140b531b-f63d-42aa-bbc6-62059f6b7aba-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw\" (UID: \"140b531b-f63d-42aa-bbc6-62059f6b7aba\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw" Nov 03 20:53:27.421857 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:27.421817 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd8hr\" (UniqueName: \"kubernetes.io/projected/140b531b-f63d-42aa-bbc6-62059f6b7aba-kube-api-access-xd8hr\") pod \"payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw\" (UID: \"140b531b-f63d-42aa-bbc6-62059f6b7aba\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw" Nov 03 20:53:27.656215 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:27.656122 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw" Nov 03 20:53:28.073578 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:28.073546 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw"] Nov 03 20:53:28.078203 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:53:28.078160 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod140b531b_f63d_42aa_bbc6_62059f6b7aba.slice/crio-f98b65e111ddfc0d52058026f195f01d83dbeffa974b54e04c8bcfa9aab0106f WatchSource:0}: Error finding container f98b65e111ddfc0d52058026f195f01d83dbeffa974b54e04c8bcfa9aab0106f: Status 404 returned error can't find the container with id f98b65e111ddfc0d52058026f195f01d83dbeffa974b54e04c8bcfa9aab0106f Nov 03 20:53:28.109175 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:28.099832 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:53:28.109175 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:28.099903 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" podUID="0df130a3-4a40-4837-8211-cfc0ed20d0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 20:53:28.481135 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:28.481040 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw" event={"ID":"140b531b-f63d-42aa-bbc6-62059f6b7aba","Type":"ContainerStarted","Data":"f0d94138dc8d402716fe206b02410e528179e93b8e70619b6f9ebf40140b914a"} Nov 03 20:53:28.481135 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:28.481093 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw" event={"ID":"140b531b-f63d-42aa-bbc6-62059f6b7aba","Type":"ContainerStarted","Data":"f98b65e111ddfc0d52058026f195f01d83dbeffa974b54e04c8bcfa9aab0106f"} Nov 03 20:53:28.481383 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:28.481285 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw" Nov 03 20:53:28.620357 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:28.619800 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw" podStartSLOduration=2.619778474 podStartE2EDuration="2.619778474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:28.60891577 +0000 UTC m=+1480.166906296" watchObservedRunningTime="2026-11-03 20:53:28.619778474 +0000 UTC m=+1480.177768974" Nov 03 20:53:29.105360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:29.101759 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:53:29.105360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:29.101824 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" podUID="0df130a3-4a40-4837-8211-cfc0ed20d0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 20:53:29.776784 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:29.776688 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k"] Nov 03 20:53:29.783734 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:29.783708 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k" Nov 03 20:53:29.838911 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:29.838885 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k"] Nov 03 20:53:29.844675 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:29.844646 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/333cb4b6-4674-462d-a66d-a00198468cb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k\" (UID: \"333cb4b6-4674-462d-a66d-a00198468cb7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k" Nov 03 20:53:29.844843 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:29.844823 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjxjm\" (UniqueName: \"kubernetes.io/projected/333cb4b6-4674-462d-a66d-a00198468cb7-kube-api-access-rjxjm\") pod \"payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k\" (UID: \"333cb4b6-4674-462d-a66d-a00198468cb7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k" Nov 03 20:53:29.881113 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:29.881082 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e3e5cd62\"" Nov 03 20:53:29.881297 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:29.881208 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e3e5cd62-dockercfg-bx96j\"" Nov 03 20:53:29.946058 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:29.946018 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjxjm\" (UniqueName: \"kubernetes.io/projected/333cb4b6-4674-462d-a66d-a00198468cb7-kube-api-access-rjxjm\") pod \"payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k\" (UID: \"333cb4b6-4674-462d-a66d-a00198468cb7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k" Nov 03 20:53:29.946272 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:29.946256 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/333cb4b6-4674-462d-a66d-a00198468cb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k\" (UID: \"333cb4b6-4674-462d-a66d-a00198468cb7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k" Nov 03 20:53:29.949894 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:29.947185 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/333cb4b6-4674-462d-a66d-a00198468cb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k\" (UID: \"333cb4b6-4674-462d-a66d-a00198468cb7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k" Nov 03 20:53:30.122358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:30.111755 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:53:30.122358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:30.111817 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" podUID="0df130a3-4a40-4837-8211-cfc0ed20d0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 20:53:30.237358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:30.236403 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjxjm\" (UniqueName: \"kubernetes.io/projected/333cb4b6-4674-462d-a66d-a00198468cb7-kube-api-access-rjxjm\") pod \"payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k\" (UID: \"333cb4b6-4674-462d-a66d-a00198468cb7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k" Nov 03 20:53:30.431889 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:30.430389 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k" Nov 03 20:53:30.835671 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:30.835627 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k"] Nov 03 20:53:30.839522 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:53:30.839495 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod333cb4b6_4674_462d_a66d_a00198468cb7.slice/crio-0c9e198317a24d1bcf58b44b1dbefc0cbd1ae9efad9d1368653f9b212819730d WatchSource:0}: Error finding container 0c9e198317a24d1bcf58b44b1dbefc0cbd1ae9efad9d1368653f9b212819730d: Status 404 returned error can't find the container with id 0c9e198317a24d1bcf58b44b1dbefc0cbd1ae9efad9d1368653f9b212819730d Nov 03 20:53:31.101480 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:31.100902 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:53:31.101480 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:31.100965 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" podUID="0df130a3-4a40-4837-8211-cfc0ed20d0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 20:53:31.559613 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:31.559567 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k" event={"ID":"333cb4b6-4674-462d-a66d-a00198468cb7","Type":"ContainerStarted","Data":"cc07146b9913000cea16d6fc2bcfbb1b6cda93a6948b652b20747ce2d2e55f67"} Nov 03 20:53:31.560016 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:31.559630 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k" event={"ID":"333cb4b6-4674-462d-a66d-a00198468cb7","Type":"ContainerStarted","Data":"0c9e198317a24d1bcf58b44b1dbefc0cbd1ae9efad9d1368653f9b212819730d"} Nov 03 20:53:31.560016 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:31.559663 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k" Nov 03 20:53:31.669717 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:31.669659 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k" podStartSLOduration=2.669644153 podStartE2EDuration="2.669644153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:31.666893802 +0000 UTC m=+1483.224884318" watchObservedRunningTime="2026-11-03 20:53:31.669644153 +0000 UTC m=+1483.227634651" Nov 03 20:53:32.097626 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:32.097586 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:53:32.097818 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:32.097661 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" podUID="0df130a3-4a40-4837-8211-cfc0ed20d0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 20:53:33.105627 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:33.105592 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:33.106434 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:33.106303 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:33.110675 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:33.110239 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:53:37.476003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:37.475975 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-43693d42-844854c99c-wkh7b" Nov 03 20:53:37.476407 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:37.476083 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w" Nov 03 20:53:38.810807 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.810762 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn"] Nov 03 20:53:38.815019 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.815000 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:38.856044 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.856009 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:38.856216 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.856063 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d81ee18-7c91-43c5-b17c-4c42835b431b-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:38.856216 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.856209 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:38.856396 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.856250 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:38.856484 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.856386 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:38.856600 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.856576 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:38.856644 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.856608 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpsfn\" (UniqueName: \"kubernetes.io/projected/0d81ee18-7c91-43c5-b17c-4c42835b431b-kube-api-access-rpsfn\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:38.856731 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.856714 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:38.856821 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.856806 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:38.916247 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.916218 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn"] Nov 03 20:53:38.958047 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.958015 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:38.958246 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.958077 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:38.958246 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.958111 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d81ee18-7c91-43c5-b17c-4c42835b431b-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:38.958246 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.958210 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:38.958490 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.958250 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:38.958490 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.958288 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:38.958490 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.958458 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:38.958490 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.958486 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpsfn\" (UniqueName: \"kubernetes.io/projected/0d81ee18-7c91-43c5-b17c-4c42835b431b-kube-api-access-rpsfn\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:38.960088 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.959006 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:38.960088 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.959178 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:38.960088 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.959200 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:38.960088 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.959491 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:38.960088 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.959534 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:38.960088 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.960016 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d81ee18-7c91-43c5-b17c-4c42835b431b-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:38.961377 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.961352 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:38.961941 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:38.961915 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:39.012162 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:39.012128 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:39.012817 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:39.012794 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpsfn\" (UniqueName: \"kubernetes.io/projected/0d81ee18-7c91-43c5-b17c-4c42835b431b-kube-api-access-rpsfn\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:39.129354 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:39.129248 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:39.566373 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:39.564081 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw" Nov 03 20:53:39.607243 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:39.607212 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn"] Nov 03 20:53:39.610096 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:53:39.610067 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d81ee18_7c91_43c5_b17c_4c42835b431b.slice/crio-eb79a0ea2963dd1e8fc8c6ae01cc7df2ed617c011d09eec3eaa179fb7d14606b WatchSource:0}: Error finding container eb79a0ea2963dd1e8fc8c6ae01cc7df2ed617c011d09eec3eaa179fb7d14606b: Status 404 returned error can't find the container with id eb79a0ea2963dd1e8fc8c6ae01cc7df2ed617c011d09eec3eaa179fb7d14606b Nov 03 20:53:39.613943 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:39.613868 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:53:39.614047 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:39.613986 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:53:39.614047 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:39.614033 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:53:39.626290 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:39.626259 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" event={"ID":"0d81ee18-7c91-43c5-b17c-4c42835b431b","Type":"ContainerStarted","Data":"eb79a0ea2963dd1e8fc8c6ae01cc7df2ed617c011d09eec3eaa179fb7d14606b"} Nov 03 20:53:40.531223 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.531185 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j"] Nov 03 20:53:40.535783 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.535763 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.611193 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.611153 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b9gd\" (UniqueName: \"kubernetes.io/projected/28a8231a-a95b-49e6-9c78-d872d17fd6cb-kube-api-access-5b9gd\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.611193 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.611199 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.611512 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.611388 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.611512 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.611459 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.611627 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.611598 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.611765 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.611744 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.611831 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.611786 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.611907 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.611890 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.611962 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.611925 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.641917 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.641872 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" event={"ID":"0d81ee18-7c91-43c5-b17c-4c42835b431b","Type":"ContainerStarted","Data":"971ddc644f48045a94c34fc754b920bb806f6408dbda32d15734654acf8deb04"} Nov 03 20:53:40.649645 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.649608 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j"] Nov 03 20:53:40.718413 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.715171 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.718413 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.715223 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.718413 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.715286 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.718413 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.715305 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.718413 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.715366 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b9gd\" (UniqueName: \"kubernetes.io/projected/28a8231a-a95b-49e6-9c78-d872d17fd6cb-kube-api-access-5b9gd\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.718413 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.715391 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.718413 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.715567 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.718413 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.715648 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.718413 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.715688 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.718413 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.716385 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.718413 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.716583 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.718413 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.717495 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.718413 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.717835 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.720505 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.720475 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.723584 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.723554 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.725263 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.725238 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.771800 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.771539 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.771800 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.771663 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b9gd\" (UniqueName: \"kubernetes.io/projected/28a8231a-a95b-49e6-9c78-d872d17fd6cb-kube-api-access-5b9gd\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:40.776458 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.776410 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" podStartSLOduration=2.776396967 podStartE2EDuration="2.776396967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:40.771774961 +0000 UTC m=+1492.329765483" watchObservedRunningTime="2026-11-03 20:53:40.776396967 +0000 UTC m=+1492.334387477" Nov 03 20:53:40.851015 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:40.850980 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:41.130260 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:41.130163 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:41.131951 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:41.131906 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:53:41.132120 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:41.131962 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" podUID="0d81ee18-7c91-43c5-b17c-4c42835b431b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 20:53:41.314354 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:41.312369 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j"] Nov 03 20:53:41.321879 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:53:41.321840 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a8231a_a95b_49e6_9c78_d872d17fd6cb.slice/crio-47c64fd1494c7e773e34f8e0496e32deee0c84da8596b28916d59a5f389d7d9b WatchSource:0}: Error finding container 47c64fd1494c7e773e34f8e0496e32deee0c84da8596b28916d59a5f389d7d9b: Status 404 returned error can't find the container with id 47c64fd1494c7e773e34f8e0496e32deee0c84da8596b28916d59a5f389d7d9b Nov 03 20:53:41.325160 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:41.325122 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:53:41.325289 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:41.325208 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:53:41.325289 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:41.325243 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:53:41.651488 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:41.651451 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" event={"ID":"28a8231a-a95b-49e6-9c78-d872d17fd6cb","Type":"ContainerStarted","Data":"1a8386cb970ec6728645b685d0ea191d47f14f9ded64e64ddaa9d145a0079655"} Nov 03 20:53:41.651973 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:41.651522 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" event={"ID":"28a8231a-a95b-49e6-9c78-d872d17fd6cb","Type":"ContainerStarted","Data":"47c64fd1494c7e773e34f8e0496e32deee0c84da8596b28916d59a5f389d7d9b"} Nov 03 20:53:41.766003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:41.765887 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" podStartSLOduration=1.765869655 podStartE2EDuration="1.765869655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:41.763671762 +0000 UTC m=+1493.321662283" watchObservedRunningTime="2026-11-03 20:53:41.765869655 +0000 UTC m=+1493.323860155" Nov 03 20:53:41.851254 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:41.851224 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:42.136359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:42.132805 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:53:42.136359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:42.132868 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" podUID="0d81ee18-7c91-43c5-b17c-4c42835b431b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 20:53:42.568847 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:42.568820 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k" Nov 03 20:53:42.854143 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:42.853902 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 20:53:42.854143 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:42.853984 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" podUID="28a8231a-a95b-49e6-9c78-d872d17fd6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 20:53:43.139420 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:43.138825 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:53:43.139420 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:43.138886 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" podUID="0d81ee18-7c91-43c5-b17c-4c42835b431b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 20:53:43.852092 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:43.852055 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 20:53:43.852278 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:43.852116 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" podUID="28a8231a-a95b-49e6-9c78-d872d17fd6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 20:53:44.130420 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:44.130035 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:53:44.130420 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:44.130099 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" podUID="0d81ee18-7c91-43c5-b17c-4c42835b431b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 20:53:44.851966 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:44.851937 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 20:53:44.852198 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:44.852177 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" podUID="28a8231a-a95b-49e6-9c78-d872d17fd6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 20:53:45.134420 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:45.133838 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:53:45.134420 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:45.133902 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" podUID="0d81ee18-7c91-43c5-b17c-4c42835b431b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 20:53:45.851922 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:45.851879 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 20:53:45.852109 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:45.851954 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" podUID="28a8231a-a95b-49e6-9c78-d872d17fd6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 20:53:46.129972 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:46.129885 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:53:46.130122 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:46.129957 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" podUID="0d81ee18-7c91-43c5-b17c-4c42835b431b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 20:53:46.852166 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:46.852133 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 20:53:46.852585 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:46.852190 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" podUID="28a8231a-a95b-49e6-9c78-d872d17fd6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 20:53:47.244079 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:47.243988 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:47.245019 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:47.244985 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:47.245707 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:47.245683 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:53:47.852119 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:47.852089 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 20:53:47.852301 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:47.852136 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" podUID="28a8231a-a95b-49e6-9c78-d872d17fd6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 20:53:48.124562 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:48.124486 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc"] Nov 03 20:53:48.128991 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:48.128967 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc" Nov 03 20:53:48.178555 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:48.178526 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-43693d42-serving-cert\"" Nov 03 20:53:48.183012 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:48.182986 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc"] Nov 03 20:53:48.224445 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:48.224410 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf717fb-3da6-4351-be0a-392cbc4434f2-openshift-service-ca\") pod \"maas-api-e2e-ait-43693d42-6786657c56-khlxc\" (UID: \"baf717fb-3da6-4351-be0a-392cbc4434f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc" Nov 03 20:53:48.224683 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:48.224660 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baf717fb-3da6-4351-be0a-392cbc4434f2-metrics-certs\") pod \"maas-api-e2e-ait-43693d42-6786657c56-khlxc\" (UID: \"baf717fb-3da6-4351-be0a-392cbc4434f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc" Nov 03 20:53:48.224882 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:48.224861 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baf717fb-3da6-4351-be0a-392cbc4434f2-maas-api-tls\") pod \"maas-api-e2e-ait-43693d42-6786657c56-khlxc\" (UID: \"baf717fb-3da6-4351-be0a-392cbc4434f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc" Nov 03 20:53:48.224998 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:48.224974 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg5rn\" (UniqueName: \"kubernetes.io/projected/baf717fb-3da6-4351-be0a-392cbc4434f2-kube-api-access-jg5rn\") pod \"maas-api-e2e-ait-43693d42-6786657c56-khlxc\" (UID: \"baf717fb-3da6-4351-be0a-392cbc4434f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc" Nov 03 20:53:48.326897 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:48.326864 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baf717fb-3da6-4351-be0a-392cbc4434f2-metrics-certs\") pod \"maas-api-e2e-ait-43693d42-6786657c56-khlxc\" (UID: \"baf717fb-3da6-4351-be0a-392cbc4434f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc" Nov 03 20:53:48.327050 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:48.326990 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baf717fb-3da6-4351-be0a-392cbc4434f2-maas-api-tls\") pod \"maas-api-e2e-ait-43693d42-6786657c56-khlxc\" (UID: \"baf717fb-3da6-4351-be0a-392cbc4434f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc" Nov 03 20:53:48.327050 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:48.327034 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg5rn\" (UniqueName: \"kubernetes.io/projected/baf717fb-3da6-4351-be0a-392cbc4434f2-kube-api-access-jg5rn\") pod \"maas-api-e2e-ait-43693d42-6786657c56-khlxc\" (UID: \"baf717fb-3da6-4351-be0a-392cbc4434f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc" Nov 03 20:53:48.327143 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:48.327076 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf717fb-3da6-4351-be0a-392cbc4434f2-openshift-service-ca\") pod \"maas-api-e2e-ait-43693d42-6786657c56-khlxc\" (UID: \"baf717fb-3da6-4351-be0a-392cbc4434f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc" Nov 03 20:53:48.327196 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:53:48.327155 4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-serving-cert: secret "maas-api-e2e-ait-43693d42-serving-cert" not found Nov 03 20:53:48.327253 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:53:48.327233 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baf717fb-3da6-4351-be0a-392cbc4434f2-maas-api-tls podName:baf717fb-3da6-4351-be0a-392cbc4434f2 nodeName:}" failed. No retries permitted until 2026-11-03 20:53:48.827210672 +0000 UTC m=+1500.385201163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/baf717fb-3da6-4351-be0a-392cbc4434f2-maas-api-tls") pod "maas-api-e2e-ait-43693d42-6786657c56-khlxc" (UID: "baf717fb-3da6-4351-be0a-392cbc4434f2") : secret "maas-api-e2e-ait-43693d42-serving-cert" not found Nov 03 20:53:48.327781 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:48.327765 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf717fb-3da6-4351-be0a-392cbc4434f2-openshift-service-ca\") pod \"maas-api-e2e-ait-43693d42-6786657c56-khlxc\" (UID: \"baf717fb-3da6-4351-be0a-392cbc4434f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc" Nov 03 20:53:48.329609 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:48.329590 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baf717fb-3da6-4351-be0a-392cbc4434f2-metrics-certs\") pod \"maas-api-e2e-ait-43693d42-6786657c56-khlxc\" (UID: \"baf717fb-3da6-4351-be0a-392cbc4434f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc" Nov 03 20:53:48.382691 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:48.382634 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg5rn\" (UniqueName: \"kubernetes.io/projected/baf717fb-3da6-4351-be0a-392cbc4434f2-kube-api-access-jg5rn\") pod \"maas-api-e2e-ait-43693d42-6786657c56-khlxc\" (UID: \"baf717fb-3da6-4351-be0a-392cbc4434f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc" Nov 03 20:53:48.817583 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:48.817553 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 20:53:48.819053 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:48.819034 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 20:53:48.841812 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:48.841774 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baf717fb-3da6-4351-be0a-392cbc4434f2-maas-api-tls\") pod \"maas-api-e2e-ait-43693d42-6786657c56-khlxc\" (UID: \"baf717fb-3da6-4351-be0a-392cbc4434f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc" Nov 03 20:53:48.844756 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:48.844731 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baf717fb-3da6-4351-be0a-392cbc4434f2-maas-api-tls\") pod \"maas-api-e2e-ait-43693d42-6786657c56-khlxc\" (UID: \"baf717fb-3da6-4351-be0a-392cbc4434f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc" Nov 03 20:53:48.886799 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:48.886759 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:48.887585 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:48.887564 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:48.890757 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:48.890738 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:53:49.045907 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:49.045879 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc" Nov 03 20:53:49.428218 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:49.428127 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc"] Nov 03 20:53:49.438404 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:53:49.438369 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf717fb_3da6_4351_be0a_392cbc4434f2.slice/crio-effa9e4ae0dc94bf0ecc11380e065a3939a5aa53d3ea2c5cc4f9f19df87471e5 WatchSource:0}: Error finding container effa9e4ae0dc94bf0ecc11380e065a3939a5aa53d3ea2c5cc4f9f19df87471e5: Status 404 returned error can't find the container with id effa9e4ae0dc94bf0ecc11380e065a3939a5aa53d3ea2c5cc4f9f19df87471e5 Nov 03 20:53:49.579569 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:49.579536 4493 scope.go:122] "RemoveContainer" containerID="a7995bb6789c04d6541cbcc96ca030fec43b65b830e669181976a0716a2a24a3" Nov 03 20:53:49.787152 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:49.787073 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc" event={"ID":"baf717fb-3da6-4351-be0a-392cbc4434f2","Type":"ContainerStarted","Data":"effa9e4ae0dc94bf0ecc11380e065a3939a5aa53d3ea2c5cc4f9f19df87471e5"} Nov 03 20:53:50.795540 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:50.795493 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc" event={"ID":"baf717fb-3da6-4351-be0a-392cbc4434f2","Type":"ContainerStarted","Data":"5e2de5e0287eb2e72f3df7589b173d3d7875a1bfecd39f8395196bb9291723d6"} Nov 03 20:53:50.906088 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:50.906034 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc" podStartSLOduration=2.077637392 podStartE2EDuration="2.906020793s" totalImagesPullingTime="828.383401ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:50.902630056 +0000 UTC m=+1502.460620556" watchObservedRunningTime="2026-11-03 20:53:50.906020793 +0000 UTC m=+1502.464011292" Nov 03 20:53:51.747120 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:53:51.747082 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:53:51.802594 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:51.802564 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc" Nov 03 20:53:55.647447 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.647416 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs"] Nov 03 20:53:55.654814 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.654797 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.662712 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.662687 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.662852 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.662837 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.662927 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.662912 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.663020 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.663000 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.663080 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.663060 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.663159 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.663142 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.663218 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.663192 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.663276 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.663265 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.663375 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.663299 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-276mf\" (UniqueName: \"kubernetes.io/projected/414267d6-2ce4-4d3d-b584-bb64ef5071f6-kube-api-access-276mf\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.709100 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.709062 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs"] Nov 03 20:53:55.765176 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.765144 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.765387 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.765192 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.765387 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.765214 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.765387 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.765233 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-276mf\" (UniqueName: \"kubernetes.io/projected/414267d6-2ce4-4d3d-b584-bb64ef5071f6-kube-api-access-276mf\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.765387 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.765276 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.765387 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.765376 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.765694 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.765461 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.765694 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.765515 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.765694 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.765605 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.765825 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.765687 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.765825 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.765746 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.766045 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.766025 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.766273 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.766253 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.766273 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.766266 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.768171 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.768139 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.769472 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.769449 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.818924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.818893 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.819080 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.819061 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-276mf\" (UniqueName: \"kubernetes.io/projected/414267d6-2ce4-4d3d-b584-bb64ef5071f6-kube-api-access-276mf\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:55.969174 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:55.969092 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:56.400372 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:56.400322 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs"] Nov 03 20:53:56.404670 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:53:56.404643 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod414267d6_2ce4_4d3d_b584_bb64ef5071f6.slice/crio-71079a6f27e023b1faec7e22db954e483bcd9a5d0e1aeec140ea69fe268f34d8 WatchSource:0}: Error finding container 71079a6f27e023b1faec7e22db954e483bcd9a5d0e1aeec140ea69fe268f34d8: Status 404 returned error can't find the container with id 71079a6f27e023b1faec7e22db954e483bcd9a5d0e1aeec140ea69fe268f34d8 Nov 03 20:53:56.407418 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:56.407386 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:53:56.407493 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:56.407452 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:53:56.407493 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:56.407485 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:53:56.844910 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:56.844872 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" event={"ID":"414267d6-2ce4-4d3d-b584-bb64ef5071f6","Type":"ContainerStarted","Data":"f308f63a3ac62f0b35ba3dfad95b0e683da3e415244c3278c7f3fa859c2430a8"} Nov 03 20:53:56.845302 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:56.844935 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" event={"ID":"414267d6-2ce4-4d3d-b584-bb64ef5071f6","Type":"ContainerStarted","Data":"71079a6f27e023b1faec7e22db954e483bcd9a5d0e1aeec140ea69fe268f34d8"} Nov 03 20:53:56.957526 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:56.957473 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" podStartSLOduration=1.9574566789999999 podStartE2EDuration="1.957456679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:56.95375372 +0000 UTC m=+1508.511744219" watchObservedRunningTime="2026-11-03 20:53:56.957456679 +0000 UTC m=+1508.515447194" Nov 03 20:53:56.969218 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:56.969194 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:53:57.815110 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:57.815080 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc" Nov 03 20:53:57.970861 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:57.970832 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:53:57.971313 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:57.970900 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" podUID="414267d6-2ce4-4d3d-b584-bb64ef5071f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 20:53:58.969400 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:58.969369 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:53:58.969568 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:58.969425 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" podUID="414267d6-2ce4-4d3d-b584-bb64ef5071f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 20:53:59.970410 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:59.970378 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:53:59.970802 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:53:59.970429 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" podUID="414267d6-2ce4-4d3d-b584-bb64ef5071f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 20:54:00.969697 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:00.969658 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:54:00.969908 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:00.969734 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" podUID="414267d6-2ce4-4d3d-b584-bb64ef5071f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 20:54:01.970418 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:01.970385 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:54:01.970805 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:01.970442 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" podUID="414267d6-2ce4-4d3d-b584-bb64ef5071f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 20:54:02.973160 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:02.971545 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:54:02.973160 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:02.971624 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" podUID="414267d6-2ce4-4d3d-b584-bb64ef5071f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 20:54:03.974852 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:03.974819 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:54:03.975369 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:03.975348 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:54:03.975983 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:03.975968 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 20:54:08.998691 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:08.997502 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:09.031031 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:09.030988 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:09.055540 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:09.055489 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:09.069058 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:09.066946 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:09.269419 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:09.267437 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded" Nov 03 20:54:09.520527 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:09.520425 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 20:54:09.569812 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:09.569770 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:09.698563 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:09.698506 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:09.699785 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:09.699743 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:54:09.999497 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:09.999434 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:10.000928 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:10.000900 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:54:10.051678 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:10.051623 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 03 20:54:10.054534 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:10.053854 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:54:10.069381 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:10.069231 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:54:10.071440 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:10.071314 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 03 20:54:10.110371 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:10.109831 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:54:10.110371 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:10.109921 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:54:10.344361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:10.327439 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:10.521672 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:10.521631 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 20:54:10.619597 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:10.532502 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:54:10.619597 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:10.598287 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 20:54:10.701526 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:10.701480 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:10.777379 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:10.768507 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:11.025100 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:10.981856 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:54:11.063358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:11.060484 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:54:11.063358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:11.060560 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:54:11.063358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:11.060615 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:11.065003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:11.027082 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:11.112359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:11.109436 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:54:11.112359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:11.109579 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 03 20:54:11.238114 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:11.230862 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:54:11.435996 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:11.435967 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:54:11.525783 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:11.525742 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:11.633360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:11.586706 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 20:54:11.633360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:11.600540 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:11.633360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:11.600677 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 20:54:11.745419 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:11.651166 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:54:11.745419 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:11.709725 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:11.751416 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:11.751390 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:54:11.770352 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:11.768854 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:11.817361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:11.800704 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 20:54:11.871964 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:11.871935 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:54:11.872759 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:11.872736 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:54:11.873184 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:11.873168 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 20:54:11.879510 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:11.879491 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:54:11.928057 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:11.928024 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:54:17.589303 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:17.589271 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw"] Nov 03 20:54:17.593722 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:17.593695 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw" Nov 03 20:54:17.655355 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:17.653632 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw"] Nov 03 20:54:17.687385 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:17.687323 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6rnf\" (UniqueName: \"kubernetes.io/projected/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-kube-api-access-d6rnf\") pod \"maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw\" (UID: \"059ef4a2-73a5-4b8e-9446-fe080a5bb3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw" Nov 03 20:54:17.687546 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:17.687401 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-maas-api-tls\") pod \"maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw\" (UID: \"059ef4a2-73a5-4b8e-9446-fe080a5bb3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw" Nov 03 20:54:17.687546 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:17.687467 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-openshift-service-ca\") pod \"maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw\" (UID: \"059ef4a2-73a5-4b8e-9446-fe080a5bb3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw" Nov 03 20:54:17.687662 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:17.687573 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-metrics-certs\") pod \"maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw\" (UID: \"059ef4a2-73a5-4b8e-9446-fe080a5bb3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw" Nov 03 20:54:17.688820 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:17.688792 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e3e5cd62-serving-cert\"" Nov 03 20:54:17.788937 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:17.788897 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-maas-api-tls\") pod \"maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw\" (UID: \"059ef4a2-73a5-4b8e-9446-fe080a5bb3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw" Nov 03 20:54:17.789130 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:17.788949 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-openshift-service-ca\") pod \"maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw\" (UID: \"059ef4a2-73a5-4b8e-9446-fe080a5bb3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw" Nov 03 20:54:17.789130 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:17.789032 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-metrics-certs\") pod \"maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw\" (UID: \"059ef4a2-73a5-4b8e-9446-fe080a5bb3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw" Nov 03 20:54:17.789130 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:54:17.789053 4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-serving-cert: secret "maas-api-e2e-mt-e3e5cd62-serving-cert" not found Nov 03 20:54:17.789293 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:54:17.789139 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-maas-api-tls podName:059ef4a2-73a5-4b8e-9446-fe080a5bb3d7 nodeName:}" failed. No retries permitted until 2026-11-03 20:54:18.289117623 +0000 UTC m=+1529.847108114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-maas-api-tls") pod "maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw" (UID: "059ef4a2-73a5-4b8e-9446-fe080a5bb3d7") : secret "maas-api-e2e-mt-e3e5cd62-serving-cert" not found Nov 03 20:54:17.789742 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:17.789446 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6rnf\" (UniqueName: \"kubernetes.io/projected/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-kube-api-access-d6rnf\") pod \"maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw\" (UID: \"059ef4a2-73a5-4b8e-9446-fe080a5bb3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw" Nov 03 20:54:17.789742 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:17.789723 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-openshift-service-ca\") pod \"maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw\" (UID: \"059ef4a2-73a5-4b8e-9446-fe080a5bb3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw" Nov 03 20:54:17.792106 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:17.792085 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-metrics-certs\") pod \"maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw\" (UID: \"059ef4a2-73a5-4b8e-9446-fe080a5bb3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw" Nov 03 20:54:17.854383 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:17.852460 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6rnf\" (UniqueName: \"kubernetes.io/projected/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-kube-api-access-d6rnf\") pod \"maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw\" (UID: \"059ef4a2-73a5-4b8e-9446-fe080a5bb3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw" Nov 03 20:54:18.301715 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:18.301681 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-maas-api-tls\") pod \"maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw\" (UID: \"059ef4a2-73a5-4b8e-9446-fe080a5bb3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw" Nov 03 20:54:18.304548 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:18.304520 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-maas-api-tls\") pod \"maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw\" (UID: \"059ef4a2-73a5-4b8e-9446-fe080a5bb3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw" Nov 03 20:54:18.512215 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:18.512173 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw" Nov 03 20:54:18.916034 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:18.915960 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw"] Nov 03 20:54:18.921020 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:54:18.920975 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059ef4a2_73a5_4b8e_9446_fe080a5bb3d7.slice/crio-573e952275375968abeaefe70e28b2844c4d99ac28c2294ebee90a6e125b5f16 WatchSource:0}: Error finding container 573e952275375968abeaefe70e28b2844c4d99ac28c2294ebee90a6e125b5f16: Status 404 returned error can't find the container with id 573e952275375968abeaefe70e28b2844c4d99ac28c2294ebee90a6e125b5f16 Nov 03 20:54:19.899491 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:19.899453 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw" event={"ID":"059ef4a2-73a5-4b8e-9446-fe080a5bb3d7","Type":"ContainerStarted","Data":"573e952275375968abeaefe70e28b2844c4d99ac28c2294ebee90a6e125b5f16"} Nov 03 20:54:20.907696 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:20.907653 4493 generic.go:356] "Generic (PLEG): container finished" podID="283bf8dd-6d9c-46b5-9bc1-3e88567d3af1" containerID="0b55a24f7062046592a97b6e3d19b652f78dc88847b1bb4386d02dcf6eee2c24" exitCode=0 Nov 03 20:54:20.908222 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:20.907726 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" event={"ID":"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1","Type":"ContainerDied","Data":"0b55a24f7062046592a97b6e3d19b652f78dc88847b1bb4386d02dcf6eee2c24"} Nov 03 20:54:20.908296 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:20.908265 4493 scope.go:122] "RemoveContainer" containerID="0b55a24f7062046592a97b6e3d19b652f78dc88847b1bb4386d02dcf6eee2c24" Nov 03 20:54:20.909748 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:20.909718 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw" event={"ID":"059ef4a2-73a5-4b8e-9446-fe080a5bb3d7","Type":"ContainerStarted","Data":"ab1a9d65db830f1957ef9ca1eb6943355a428adbc6775f37cf68e289194d3362"} Nov 03 20:54:20.909866 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:20.909803 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw" Nov 03 20:54:20.910259 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:20.910228 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:54:20.910354 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:20.910319 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:54:20.910409 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:20.910385 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:54:20.912246 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:20.912212 4493 generic.go:356] "Generic (PLEG): container finished" podID="d8170541-ea8d-400b-9636-52a81628985f" containerID="39298848a37ab689208b4dc15abf27c2ecad7c612df8a1413fbb1a83f6a7684e" exitCode=0 Nov 03 20:54:20.912360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:20.912240 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" event={"ID":"d8170541-ea8d-400b-9636-52a81628985f","Type":"ContainerDied","Data":"39298848a37ab689208b4dc15abf27c2ecad7c612df8a1413fbb1a83f6a7684e"} Nov 03 20:54:20.912638 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:20.912617 4493 scope.go:122] "RemoveContainer" containerID="39298848a37ab689208b4dc15abf27c2ecad7c612df8a1413fbb1a83f6a7684e" Nov 03 20:54:20.913945 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:20.913915 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:54:20.914042 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:20.913996 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:54:20.914112 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:20.914052 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:54:21.307813 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:21.307763 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw" podStartSLOduration=3.146064008 podStartE2EDuration="4.307745047s" totalImagesPullingTime="1.161681039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:21.30644151 +0000 UTC m=+1532.864432036" watchObservedRunningTime="2026-11-03 20:54:21.307745047 +0000 UTC m=+1532.865735546" Nov 03 20:54:21.776326 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:54:21.776240 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:54:21.939351 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:21.939303 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" event={"ID":"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1","Type":"ContainerStarted","Data":"ad893a1354c8507bfec4ff6d9dec8734958079da8435b2a8d82a8d8d483d1685"} Nov 03 20:54:21.941619 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:21.941593 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" event={"ID":"d8170541-ea8d-400b-9636-52a81628985f","Type":"ContainerStarted","Data":"8bcafe6ab04dc24fb3eb68581469e3442507d8b9f22700711c803aa686a725d5"} Nov 03 20:54:22.757931 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:22.757897 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:54:22.758122 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:22.758019 4493 patch_prober.go:29] interesting pod/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:54:22.758122 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:22.758063 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" podUID="283bf8dd-6d9c-46b5-9bc1-3e88567d3af1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 20:54:22.924406 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:22.924377 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:54:22.933631 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:22.933596 4493 patch_prober.go:29] interesting pod/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 20:54:22.933904 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:22.933872 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" podUID="d8170541-ea8d-400b-9636-52a81628985f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 20:54:23.759976 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:23.759944 4493 patch_prober.go:29] interesting pod/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:54:23.760481 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:23.759996 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" podUID="283bf8dd-6d9c-46b5-9bc1-3e88567d3af1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 20:54:23.932358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:23.921050 4493 patch_prober.go:29] interesting pod/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 20:54:23.932358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:23.921121 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" podUID="d8170541-ea8d-400b-9636-52a81628985f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 20:54:24.757933 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:24.757892 4493 patch_prober.go:29] interesting pod/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:54:24.758143 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:24.757956 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" podUID="283bf8dd-6d9c-46b5-9bc1-3e88567d3af1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 20:54:24.917396 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:24.917359 4493 patch_prober.go:29] interesting pod/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 20:54:24.918046 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:24.918014 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" podUID="d8170541-ea8d-400b-9636-52a81628985f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 20:54:25.758457 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:25.758418 4493 patch_prober.go:29] interesting pod/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:54:25.758647 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:25.758481 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" podUID="283bf8dd-6d9c-46b5-9bc1-3e88567d3af1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 20:54:25.915778 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:25.915738 4493 patch_prober.go:29] interesting pod/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 20:54:25.916063 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:25.916034 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" podUID="d8170541-ea8d-400b-9636-52a81628985f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 20:54:26.757921 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:26.757884 4493 patch_prober.go:29] interesting pod/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:54:26.758324 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:26.757940 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" podUID="283bf8dd-6d9c-46b5-9bc1-3e88567d3af1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 20:54:26.924730 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:26.924693 4493 patch_prober.go:29] interesting pod/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 20:54:26.926170 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:26.926137 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" podUID="d8170541-ea8d-400b-9636-52a81628985f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 20:54:26.948205 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:26.948180 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw" Nov 03 20:54:27.181960 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:27.181932 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:54:27.763492 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:27.763464 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:54:27.763907 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:27.763750 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:54:27.764496 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:27.764475 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:54:27.918653 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:27.918614 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:54:28.135102 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:28.135077 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:54:51.854623 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:54:51.854580 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:54:57.371711 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:57.371676 4493 generic.go:356] "Generic (PLEG): container finished" podID="9f4c678e-8285-4d16-984d-39fa208f1ae8" containerID="1a4e7de59a82a80ef3ef355cd595031cf005610156f6488827b292e41cc035ee" exitCode=0 Nov 03 20:54:57.372145 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:57.371756 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" event={"ID":"9f4c678e-8285-4d16-984d-39fa208f1ae8","Type":"ContainerDied","Data":"1a4e7de59a82a80ef3ef355cd595031cf005610156f6488827b292e41cc035ee"} Nov 03 20:54:57.372306 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:57.372289 4493 scope.go:122] "RemoveContainer" containerID="1a4e7de59a82a80ef3ef355cd595031cf005610156f6488827b292e41cc035ee" Nov 03 20:54:57.374063 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:57.374032 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:54:57.374154 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:57.374105 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:54:57.374154 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:57.374140 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:54:58.380919 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:58.380882 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" event={"ID":"9f4c678e-8285-4d16-984d-39fa208f1ae8","Type":"ContainerStarted","Data":"25a222b086d47718c2e12b83bcfb777181cc5c315736b175d023640e33330fbe"} Nov 03 20:54:59.330590 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:59.330552 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:54:59.330853 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:59.330830 4493 patch_prober.go:29] interesting pod/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 20:54:59.330910 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:54:59.330887 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" podUID="9f4c678e-8285-4d16-984d-39fa208f1ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 20:55:00.331271 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:00.331209 4493 patch_prober.go:29] interesting pod/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 20:55:00.331271 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:00.331258 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" podUID="9f4c678e-8285-4d16-984d-39fa208f1ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 20:55:01.330611 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:01.330581 4493 patch_prober.go:29] interesting pod/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 20:55:01.330803 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:01.330696 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" podUID="9f4c678e-8285-4d16-984d-39fa208f1ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 20:55:02.331133 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:02.331103 4493 patch_prober.go:29] interesting pod/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 20:55:02.331587 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:02.331156 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" podUID="9f4c678e-8285-4d16-984d-39fa208f1ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 20:55:03.331473 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:03.331434 4493 patch_prober.go:29] interesting pod/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 20:55:03.331963 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:03.331496 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" podUID="9f4c678e-8285-4d16-984d-39fa208f1ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 20:55:04.334865 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:04.334835 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:55:04.335367 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:04.335317 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:55:04.336000 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:04.335979 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:55:09.073983 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:09.073942 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-43693d42-844854c99c-wkh7b"] Nov 03 20:55:09.189642 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:09.189606 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w"] Nov 03 20:55:14.075036 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.074992 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-43693d42-844854c99c-wkh7b" podUID="b0fec100-98d7-4570-860e-0d89ab70f264" containerName="payload-processing" containerID="cri-o://de0f4f7e19be58d0404afa37965a6bfd33e11be9ad5dc4afaac5ee056f7e98c7" gracePeriod=25 Nov 03 20:55:14.199454 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.199384 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w" podUID="2b783dd2-815d-40be-b790-4e303c07e82c" containerName="payload-pre-processing" containerID="cri-o://09243faedf52d27d133a6cc95cc6f34c38c9ca3652923301215762ec97308cd1" gracePeriod=25 Nov 03 20:55:14.458578 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.458056 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-43693d42-844854c99c-wkh7b" Nov 03 20:55:14.548689 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.548661 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0fec100-98d7-4570-860e-0d89ab70f264-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0fec100-98d7-4570-860e-0d89ab70f264-plugins-config-volume\") pod \"b0fec100-98d7-4570-860e-0d89ab70f264\" (UID: \"b0fec100-98d7-4570-860e-0d89ab70f264\") " Nov 03 20:55:14.548910 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.548890 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0fec100-98d7-4570-860e-0d89ab70f264-kube-api-access-vpsrg\" (UniqueName: \"kubernetes.io/projected/b0fec100-98d7-4570-860e-0d89ab70f264-kube-api-access-vpsrg\") pod \"b0fec100-98d7-4570-860e-0d89ab70f264\" (UID: \"b0fec100-98d7-4570-860e-0d89ab70f264\") " Nov 03 20:55:14.549054 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.549036 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0fec100-98d7-4570-860e-0d89ab70f264-plugins-config-volume" pod "b0fec100-98d7-4570-860e-0d89ab70f264" (UID: "b0fec100-98d7-4570-860e-0d89ab70f264"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:14.549593 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.549535 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0fec100-98d7-4570-860e-0d89ab70f264-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:14.551097 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.551075 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0fec100-98d7-4570-860e-0d89ab70f264-kube-api-access-vpsrg" pod "b0fec100-98d7-4570-860e-0d89ab70f264" (UID: "b0fec100-98d7-4570-860e-0d89ab70f264"). InnerVolumeSpecName "kube-api-access-vpsrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:14.557328 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.557306 4493 generic.go:356] "Generic (PLEG): container finished" podID="2b783dd2-815d-40be-b790-4e303c07e82c" containerID="09243faedf52d27d133a6cc95cc6f34c38c9ca3652923301215762ec97308cd1" exitCode=0 Nov 03 20:55:14.559202 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.559182 4493 generic.go:356] "Generic (PLEG): container finished" podID="b0fec100-98d7-4570-860e-0d89ab70f264" containerID="de0f4f7e19be58d0404afa37965a6bfd33e11be9ad5dc4afaac5ee056f7e98c7" exitCode=0 Nov 03 20:55:14.559371 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.559310 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-43693d42-844854c99c-wkh7b" Nov 03 20:55:14.566275 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.566246 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w" event={"ID":"2b783dd2-815d-40be-b790-4e303c07e82c","Type":"ContainerDied","Data":"09243faedf52d27d133a6cc95cc6f34c38c9ca3652923301215762ec97308cd1"} Nov 03 20:55:14.566369 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.566300 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w" event={"ID":"2b783dd2-815d-40be-b790-4e303c07e82c","Type":"ContainerDied","Data":"bac30682d1e9d25560c450370c8b9f9c1d39f65ab739385970e29924350f57b1"} Nov 03 20:55:14.566369 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.566362 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bac30682d1e9d25560c450370c8b9f9c1d39f65ab739385970e29924350f57b1" Nov 03 20:55:14.566477 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.566378 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-43693d42-844854c99c-wkh7b" event={"ID":"b0fec100-98d7-4570-860e-0d89ab70f264","Type":"ContainerDied","Data":"de0f4f7e19be58d0404afa37965a6bfd33e11be9ad5dc4afaac5ee056f7e98c7"} Nov 03 20:55:14.566477 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.566413 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-43693d42-844854c99c-wkh7b" event={"ID":"b0fec100-98d7-4570-860e-0d89ab70f264","Type":"ContainerDied","Data":"31c120d7517360437e633c8a97316eb3dc8fa8f9de7471bac30bac8d6d1d0225"} Nov 03 20:55:14.566477 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.566435 4493 scope.go:122] "RemoveContainer" containerID="de0f4f7e19be58d0404afa37965a6bfd33e11be9ad5dc4afaac5ee056f7e98c7" Nov 03 20:55:14.579089 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.579034 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w" Nov 03 20:55:14.611582 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.610135 4493 scope.go:122] "RemoveContainer" containerID="de0f4f7e19be58d0404afa37965a6bfd33e11be9ad5dc4afaac5ee056f7e98c7" Nov 03 20:55:14.611582 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:55:14.610650 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de0f4f7e19be58d0404afa37965a6bfd33e11be9ad5dc4afaac5ee056f7e98c7\": container with ID starting with de0f4f7e19be58d0404afa37965a6bfd33e11be9ad5dc4afaac5ee056f7e98c7 not found: ID does not exist" containerID="de0f4f7e19be58d0404afa37965a6bfd33e11be9ad5dc4afaac5ee056f7e98c7" Nov 03 20:55:14.611582 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.610691 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de0f4f7e19be58d0404afa37965a6bfd33e11be9ad5dc4afaac5ee056f7e98c7"} err="failed to get container status \"de0f4f7e19be58d0404afa37965a6bfd33e11be9ad5dc4afaac5ee056f7e98c7\": rpc error: code = NotFound desc = could not find container \"de0f4f7e19be58d0404afa37965a6bfd33e11be9ad5dc4afaac5ee056f7e98c7\": container with ID starting with de0f4f7e19be58d0404afa37965a6bfd33e11be9ad5dc4afaac5ee056f7e98c7 not found: ID does not exist" Nov 03 20:55:14.651911 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.651880 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpsrg\" (UniqueName: \"kubernetes.io/projected/b0fec100-98d7-4570-860e-0d89ab70f264-kube-api-access-vpsrg\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:14.752448 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.752416 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b783dd2-815d-40be-b790-4e303c07e82c-kube-api-access-sj8l5\" (UniqueName: \"kubernetes.io/projected/2b783dd2-815d-40be-b790-4e303c07e82c-kube-api-access-sj8l5\") pod \"2b783dd2-815d-40be-b790-4e303c07e82c\" (UID: \"2b783dd2-815d-40be-b790-4e303c07e82c\") " Nov 03 20:55:14.752633 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.752615 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b783dd2-815d-40be-b790-4e303c07e82c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b783dd2-815d-40be-b790-4e303c07e82c-plugins-config-volume\") pod \"2b783dd2-815d-40be-b790-4e303c07e82c\" (UID: \"2b783dd2-815d-40be-b790-4e303c07e82c\") " Nov 03 20:55:14.753008 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.752971 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b783dd2-815d-40be-b790-4e303c07e82c-plugins-config-volume" pod "2b783dd2-815d-40be-b790-4e303c07e82c" (UID: "2b783dd2-815d-40be-b790-4e303c07e82c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:14.753513 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.753491 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b783dd2-815d-40be-b790-4e303c07e82c-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:14.754788 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.754767 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b783dd2-815d-40be-b790-4e303c07e82c-kube-api-access-sj8l5" pod "2b783dd2-815d-40be-b790-4e303c07e82c" (UID: "2b783dd2-815d-40be-b790-4e303c07e82c"). InnerVolumeSpecName "kube-api-access-sj8l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:14.760170 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.760146 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-43693d42-844854c99c-wkh7b"] Nov 03 20:55:14.766114 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.766089 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-43693d42-844854c99c-wkh7b"] Nov 03 20:55:14.855035 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:14.855001 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj8l5\" (UniqueName: \"kubernetes.io/projected/2b783dd2-815d-40be-b790-4e303c07e82c-kube-api-access-sj8l5\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:15.568067 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:15.568034 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w" Nov 03 20:55:15.730866 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:15.730833 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w"] Nov 03 20:55:15.734875 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:15.734850 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-43693d42-5c6b65d6f5-ps87w"] Nov 03 20:55:16.563066 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:16.563030 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b783dd2-815d-40be-b790-4e303c07e82c" path="/var/lib/kubelet/pods/2b783dd2-815d-40be-b790-4e303c07e82c/volumes" Nov 03 20:55:16.563562 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:16.563543 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0fec100-98d7-4570-860e-0d89ab70f264" path="/var/lib/kubelet/pods/b0fec100-98d7-4570-860e-0d89ab70f264/volumes" Nov 03 20:55:21.613855 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:21.613770 4493 generic.go:356] "Generic (PLEG): container finished" podID="1f96068d-fb57-4a30-876a-e3158d210c3c" containerID="eb5d8bdef9431fa1341ba5ed49b46cbf67a0875fde27af1822fd63ba0b5c3702" exitCode=0 Nov 03 20:55:21.613855 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:21.613826 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" event={"ID":"1f96068d-fb57-4a30-876a-e3158d210c3c","Type":"ContainerDied","Data":"eb5d8bdef9431fa1341ba5ed49b46cbf67a0875fde27af1822fd63ba0b5c3702"} Nov 03 20:55:21.614293 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:21.614241 4493 scope.go:122] "RemoveContainer" containerID="eb5d8bdef9431fa1341ba5ed49b46cbf67a0875fde27af1822fd63ba0b5c3702" Nov 03 20:55:21.615955 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:21.615926 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:55:21.616025 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:21.615997 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:55:21.616065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:21.616030 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:55:21.758998 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:55:21.758948 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:55:22.622715 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:22.622685 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" event={"ID":"1f96068d-fb57-4a30-876a-e3158d210c3c","Type":"ContainerStarted","Data":"b393a9b8a28d83ec6735dc5a54e18f7073e4a9f406519cf338a769ca670c8f7c"} Nov 03 20:55:22.821520 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:22.821489 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:55:22.851551 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:22.851523 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:55:22.852600 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:22.852576 4493 patch_prober.go:29] interesting pod/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:55:22.852714 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:22.852624 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" podUID="1f96068d-fb57-4a30-876a-e3158d210c3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 20:55:23.851587 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:23.851549 4493 patch_prober.go:29] interesting pod/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:55:23.851978 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:23.851618 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" podUID="1f96068d-fb57-4a30-876a-e3158d210c3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 20:55:24.852121 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:24.852090 4493 patch_prober.go:29] interesting pod/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:55:24.852661 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:24.852156 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" podUID="1f96068d-fb57-4a30-876a-e3158d210c3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 20:55:25.852456 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:25.852426 4493 patch_prober.go:29] interesting pod/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:55:25.852885 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:25.852501 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" podUID="1f96068d-fb57-4a30-876a-e3158d210c3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 20:55:26.852489 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:26.852457 4493 patch_prober.go:29] interesting pod/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:55:26.852914 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:26.852510 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" podUID="1f96068d-fb57-4a30-876a-e3158d210c3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 20:55:27.852413 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:27.852381 4493 patch_prober.go:29] interesting pod/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:55:27.852585 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:27.852443 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" podUID="1f96068d-fb57-4a30-876a-e3158d210c3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 20:55:28.856777 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:28.856733 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:55:28.857711 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:28.857688 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:55:29.401183 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:29.401149 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw"] Nov 03 20:55:34.401744 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:34.401706 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw" podUID="140b531b-f63d-42aa-bbc6-62059f6b7aba" containerName="payload-pre-processing" containerID="cri-o://f0d94138dc8d402716fe206b02410e528179e93b8e70619b6f9ebf40140b914a" gracePeriod=25 Nov 03 20:55:34.709525 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:34.709499 4493 generic.go:356] "Generic (PLEG): container finished" podID="140b531b-f63d-42aa-bbc6-62059f6b7aba" containerID="f0d94138dc8d402716fe206b02410e528179e93b8e70619b6f9ebf40140b914a" exitCode=0 Nov 03 20:55:34.709659 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:34.709566 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw" event={"ID":"140b531b-f63d-42aa-bbc6-62059f6b7aba","Type":"ContainerDied","Data":"f0d94138dc8d402716fe206b02410e528179e93b8e70619b6f9ebf40140b914a"} Nov 03 20:55:34.709659 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:34.709618 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw" event={"ID":"140b531b-f63d-42aa-bbc6-62059f6b7aba","Type":"ContainerDied","Data":"f98b65e111ddfc0d52058026f195f01d83dbeffa974b54e04c8bcfa9aab0106f"} Nov 03 20:55:34.709659 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:34.709640 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f98b65e111ddfc0d52058026f195f01d83dbeffa974b54e04c8bcfa9aab0106f" Nov 03 20:55:34.710658 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:34.710636 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw" Nov 03 20:55:34.803774 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:34.803740 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/140b531b-f63d-42aa-bbc6-62059f6b7aba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/140b531b-f63d-42aa-bbc6-62059f6b7aba-plugins-config-volume\") pod \"140b531b-f63d-42aa-bbc6-62059f6b7aba\" (UID: \"140b531b-f63d-42aa-bbc6-62059f6b7aba\") " Nov 03 20:55:34.804086 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:34.804067 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/140b531b-f63d-42aa-bbc6-62059f6b7aba-kube-api-access-xd8hr\" (UniqueName: \"kubernetes.io/projected/140b531b-f63d-42aa-bbc6-62059f6b7aba-kube-api-access-xd8hr\") pod \"140b531b-f63d-42aa-bbc6-62059f6b7aba\" (UID: \"140b531b-f63d-42aa-bbc6-62059f6b7aba\") " Nov 03 20:55:34.804147 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:34.804085 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/140b531b-f63d-42aa-bbc6-62059f6b7aba-plugins-config-volume" pod "140b531b-f63d-42aa-bbc6-62059f6b7aba" (UID: "140b531b-f63d-42aa-bbc6-62059f6b7aba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:34.804749 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:34.804729 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/140b531b-f63d-42aa-bbc6-62059f6b7aba-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:34.806393 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:34.806374 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/140b531b-f63d-42aa-bbc6-62059f6b7aba-kube-api-access-xd8hr" pod "140b531b-f63d-42aa-bbc6-62059f6b7aba" (UID: "140b531b-f63d-42aa-bbc6-62059f6b7aba"). InnerVolumeSpecName "kube-api-access-xd8hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:34.871025 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:34.870996 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2"] Nov 03 20:55:34.871260 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:34.871237 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" podUID="d8170541-ea8d-400b-9636-52a81628985f" containerName="istio-proxy" containerID="cri-o://8bcafe6ab04dc24fb3eb68581469e3442507d8b9f22700711c803aa686a725d5" gracePeriod=30 Nov 03 20:55:34.906596 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:34.906566 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd8hr\" (UniqueName: \"kubernetes.io/projected/140b531b-f63d-42aa-bbc6-62059f6b7aba-kube-api-access-xd8hr\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:35.717782 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:35.717750 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw" Nov 03 20:55:35.875069 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:35.875030 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw"] Nov 03 20:55:35.921878 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:35.921846 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1a55d7db-7bb88854bc-zgjgw"] Nov 03 20:55:36.421315 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.421283 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg"] Nov 03 20:55:36.422592 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.422560 4493 cpu_manager.go:414] "containerMap: removing container" podUID="2b783dd2-815d-40be-b790-4e303c07e82c" containerName="payload-pre-processing" Nov 03 20:55:36.422592 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.422590 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b783dd2-815d-40be-b790-4e303c07e82c" containerName="payload-pre-processing" Nov 03 20:55:36.422807 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.422630 4493 cpu_manager.go:414] "containerMap: removing container" podUID="140b531b-f63d-42aa-bbc6-62059f6b7aba" containerName="payload-pre-processing" Nov 03 20:55:36.422807 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.422639 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="140b531b-f63d-42aa-bbc6-62059f6b7aba" containerName="payload-pre-processing" Nov 03 20:55:36.422807 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.422685 4493 cpu_manager.go:414] "containerMap: removing container" podUID="b0fec100-98d7-4570-860e-0d89ab70f264" containerName="payload-processing" Nov 03 20:55:36.422807 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.422695 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0fec100-98d7-4570-860e-0d89ab70f264" containerName="payload-processing" Nov 03 20:55:36.422964 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.422854 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="140b531b-f63d-42aa-bbc6-62059f6b7aba" containerName="payload-pre-processing" Nov 03 20:55:36.422964 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.422875 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0fec100-98d7-4570-860e-0d89ab70f264" containerName="payload-processing" Nov 03 20:55:36.422964 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.422894 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b783dd2-815d-40be-b790-4e303c07e82c" containerName="payload-pre-processing" Nov 03 20:55:36.427724 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.427696 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.478354 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.478306 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0e8374c4-openshift-default-dockercfg-cvhv4\"" Nov 03 20:55:36.533205 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.533172 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg"] Nov 03 20:55:36.541403 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.541368 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istiod-ca-cert\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.541558 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.541435 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-envoy\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.541558 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.541485 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-credential-socket\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.541703 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.541676 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-token\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.541766 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.541739 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-podinfo\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.541838 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.541786 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8c79\" (UniqueName: \"kubernetes.io/projected/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-kube-api-access-h8c79\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.541979 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.541956 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-data\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.542093 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.541995 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-workload-certs\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.542136 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.542096 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-workload-socket\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.591582 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.591554 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="140b531b-f63d-42aa-bbc6-62059f6b7aba" path="/var/lib/kubelet/pods/140b531b-f63d-42aa-bbc6-62059f6b7aba/volumes" Nov 03 20:55:36.644209 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.644172 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-credential-socket\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.644393 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.644291 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-token\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.644393 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.644376 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-podinfo\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.644553 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.644423 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8c79\" (UniqueName: \"kubernetes.io/projected/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-kube-api-access-h8c79\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.644608 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.644585 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-data\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.644664 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.644616 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-workload-certs\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.644664 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.644639 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-credential-socket\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.645258 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.645232 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-workload-certs\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.645594 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.645276 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-data\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.645594 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.645352 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-workload-socket\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.645594 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.645407 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istiod-ca-cert\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.645594 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.645450 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-envoy\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.645844 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.645684 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-workload-socket\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.645979 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.645956 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istiod-ca-cert\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.647828 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.647757 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-podinfo\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.648757 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.648734 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-envoy\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.705433 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.701712 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-token\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.705433 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.702563 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8c79\" (UniqueName: \"kubernetes.io/projected/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-kube-api-access-h8c79\") pod \"e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:36.746782 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:36.746753 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:37.185780 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:37.185751 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg"] Nov 03 20:55:37.189606 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:55:37.189578 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47bd78c_2c95_4eb1_a2a3_b151a424ebdc.slice/crio-5f1414a1468a297be4492d4570febaab03b21c8d96c25e4835e9569882b2ef0a WatchSource:0}: Error finding container 5f1414a1468a297be4492d4570febaab03b21c8d96c25e4835e9569882b2ef0a: Status 404 returned error can't find the container with id 5f1414a1468a297be4492d4570febaab03b21c8d96c25e4835e9569882b2ef0a Nov 03 20:55:37.192035 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:37.192003 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:55:37.192121 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:37.192071 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:55:37.192121 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:37.192102 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:55:37.738110 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:37.738077 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" event={"ID":"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc","Type":"ContainerStarted","Data":"5d044bb04f3c6228a831e8fbf88f5adfbab9956ac9bac3af9fbf979d3ce28463"} Nov 03 20:55:37.738316 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:37.738138 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" event={"ID":"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc","Type":"ContainerStarted","Data":"5f1414a1468a297be4492d4570febaab03b21c8d96c25e4835e9569882b2ef0a"} Nov 03 20:55:37.747454 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:37.747427 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:37.851523 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:37.851471 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" podStartSLOduration=1.851452715 podStartE2EDuration="1.851452715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:37.847263315 +0000 UTC m=+1609.405253834" watchObservedRunningTime="2026-11-03 20:55:37.851452715 +0000 UTC m=+1609.409443217" Nov 03 20:55:38.752762 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:38.752736 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:39.751607 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:39.751565 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:39.752703 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:39.752676 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:55:40.172094 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.172069 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:55:40.315853 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.315774 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-istio-envoy\") pod \"d8170541-ea8d-400b-9636-52a81628985f\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " Nov 03 20:55:40.315999 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.315950 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-workload-socket\") pod \"d8170541-ea8d-400b-9636-52a81628985f\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " Nov 03 20:55:40.315999 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.315977 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8170541-ea8d-400b-9636-52a81628985f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8170541-ea8d-400b-9636-52a81628985f-istiod-ca-cert\") pod \"d8170541-ea8d-400b-9636-52a81628985f\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " Nov 03 20:55:40.316080 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.316004 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8170541-ea8d-400b-9636-52a81628985f-kube-api-access-mlxhf\" (UniqueName: \"kubernetes.io/projected/d8170541-ea8d-400b-9636-52a81628985f-kube-api-access-mlxhf\") pod \"d8170541-ea8d-400b-9636-52a81628985f\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " Nov 03 20:55:40.316080 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.316036 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-workload-certs\") pod \"d8170541-ea8d-400b-9636-52a81628985f\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " Nov 03 20:55:40.316178 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.316070 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8170541-ea8d-400b-9636-52a81628985f-istio-token\" (UniqueName: \"kubernetes.io/projected/d8170541-ea8d-400b-9636-52a81628985f-istio-token\") pod \"d8170541-ea8d-400b-9636-52a81628985f\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " Nov 03 20:55:40.316272 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.316245 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-workload-socket" pod "d8170541-ea8d-400b-9636-52a81628985f" (UID: "d8170541-ea8d-400b-9636-52a81628985f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:40.316352 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.316279 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-istio-data\") pod \"d8170541-ea8d-400b-9636-52a81628985f\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " Nov 03 20:55:40.316405 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.316353 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-workload-certs" pod "d8170541-ea8d-400b-9636-52a81628985f" (UID: "d8170541-ea8d-400b-9636-52a81628985f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:40.316405 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.316376 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8170541-ea8d-400b-9636-52a81628985f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8170541-ea8d-400b-9636-52a81628985f-istio-podinfo\") pod \"d8170541-ea8d-400b-9636-52a81628985f\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " Nov 03 20:55:40.316678 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.316514 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-credential-socket\") pod \"d8170541-ea8d-400b-9636-52a81628985f\" (UID: \"d8170541-ea8d-400b-9636-52a81628985f\") " Nov 03 20:55:40.316678 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.316558 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8170541-ea8d-400b-9636-52a81628985f-istiod-ca-cert" pod "d8170541-ea8d-400b-9636-52a81628985f" (UID: "d8170541-ea8d-400b-9636-52a81628985f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:40.316826 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.316800 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-credential-socket" pod "d8170541-ea8d-400b-9636-52a81628985f" (UID: "d8170541-ea8d-400b-9636-52a81628985f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:40.317057 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.316994 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-istio-data" pod "d8170541-ea8d-400b-9636-52a81628985f" (UID: "d8170541-ea8d-400b-9636-52a81628985f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:40.317456 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.317434 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:40.317552 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.317461 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:40.317552 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.317476 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:40.317552 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.317493 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8170541-ea8d-400b-9636-52a81628985f-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:40.317552 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.317508 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:40.318414 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.318386 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-istio-envoy" pod "d8170541-ea8d-400b-9636-52a81628985f" (UID: "d8170541-ea8d-400b-9636-52a81628985f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:40.318652 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.318626 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8170541-ea8d-400b-9636-52a81628985f-istio-token" pod "d8170541-ea8d-400b-9636-52a81628985f" (UID: "d8170541-ea8d-400b-9636-52a81628985f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:40.318752 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.318696 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8170541-ea8d-400b-9636-52a81628985f-kube-api-access-mlxhf" pod "d8170541-ea8d-400b-9636-52a81628985f" (UID: "d8170541-ea8d-400b-9636-52a81628985f"). InnerVolumeSpecName "kube-api-access-mlxhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:40.318825 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.318808 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8170541-ea8d-400b-9636-52a81628985f-istio-podinfo" pod "d8170541-ea8d-400b-9636-52a81628985f" (UID: "d8170541-ea8d-400b-9636-52a81628985f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:55:40.419615 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.419578 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8170541-ea8d-400b-9636-52a81628985f-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:40.419615 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.419606 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlxhf\" (UniqueName: \"kubernetes.io/projected/d8170541-ea8d-400b-9636-52a81628985f-kube-api-access-mlxhf\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:40.419615 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.419618 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8170541-ea8d-400b-9636-52a81628985f-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:40.419870 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.419627 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8170541-ea8d-400b-9636-52a81628985f-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:40.761361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.761292 4493 generic.go:356] "Generic (PLEG): container finished" podID="d8170541-ea8d-400b-9636-52a81628985f" containerID="8bcafe6ab04dc24fb3eb68581469e3442507d8b9f22700711c803aa686a725d5" exitCode=0 Nov 03 20:55:40.761556 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.761364 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" Nov 03 20:55:40.761556 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.761380 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" event={"ID":"d8170541-ea8d-400b-9636-52a81628985f","Type":"ContainerDied","Data":"8bcafe6ab04dc24fb3eb68581469e3442507d8b9f22700711c803aa686a725d5"} Nov 03 20:55:40.761556 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.761446 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2" event={"ID":"d8170541-ea8d-400b-9636-52a81628985f","Type":"ContainerDied","Data":"df8a9075c3b261e34bca1ad8e874f2350629c7347b7f753eefb0bc10c2b4edaf"} Nov 03 20:55:40.761556 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.761467 4493 scope.go:122] "RemoveContainer" containerID="8bcafe6ab04dc24fb3eb68581469e3442507d8b9f22700711c803aa686a725d5" Nov 03 20:55:40.772300 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.772279 4493 scope.go:122] "RemoveContainer" containerID="39298848a37ab689208b4dc15abf27c2ecad7c612df8a1413fbb1a83f6a7684e" Nov 03 20:55:40.781749 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.781725 4493 scope.go:122] "RemoveContainer" containerID="8bcafe6ab04dc24fb3eb68581469e3442507d8b9f22700711c803aa686a725d5" Nov 03 20:55:40.782030 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:55:40.782008 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bcafe6ab04dc24fb3eb68581469e3442507d8b9f22700711c803aa686a725d5\": container with ID starting with 8bcafe6ab04dc24fb3eb68581469e3442507d8b9f22700711c803aa686a725d5 not found: ID does not exist" containerID="8bcafe6ab04dc24fb3eb68581469e3442507d8b9f22700711c803aa686a725d5" Nov 03 20:55:40.782119 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.782041 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bcafe6ab04dc24fb3eb68581469e3442507d8b9f22700711c803aa686a725d5"} err="failed to get container status \"8bcafe6ab04dc24fb3eb68581469e3442507d8b9f22700711c803aa686a725d5\": rpc error: code = NotFound desc = could not find container \"8bcafe6ab04dc24fb3eb68581469e3442507d8b9f22700711c803aa686a725d5\": container with ID starting with 8bcafe6ab04dc24fb3eb68581469e3442507d8b9f22700711c803aa686a725d5 not found: ID does not exist" Nov 03 20:55:40.782119 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.782061 4493 scope.go:122] "RemoveContainer" containerID="39298848a37ab689208b4dc15abf27c2ecad7c612df8a1413fbb1a83f6a7684e" Nov 03 20:55:40.782325 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:55:40.782309 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39298848a37ab689208b4dc15abf27c2ecad7c612df8a1413fbb1a83f6a7684e\": container with ID starting with 39298848a37ab689208b4dc15abf27c2ecad7c612df8a1413fbb1a83f6a7684e not found: ID does not exist" containerID="39298848a37ab689208b4dc15abf27c2ecad7c612df8a1413fbb1a83f6a7684e" Nov 03 20:55:40.782325 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.782345 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39298848a37ab689208b4dc15abf27c2ecad7c612df8a1413fbb1a83f6a7684e"} err="failed to get container status \"39298848a37ab689208b4dc15abf27c2ecad7c612df8a1413fbb1a83f6a7684e\": rpc error: code = NotFound desc = could not find container \"39298848a37ab689208b4dc15abf27c2ecad7c612df8a1413fbb1a83f6a7684e\": container with ID starting with 39298848a37ab689208b4dc15abf27c2ecad7c612df8a1413fbb1a83f6a7684e not found: ID does not exist" Nov 03 20:55:40.919053 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.918981 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2"] Nov 03 20:55:40.966373 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:40.966327 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-e20715ef-gw-openshift-default-77d448b849-jtnd2"] Nov 03 20:55:42.564229 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:42.564197 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8170541-ea8d-400b-9636-52a81628985f" path="/var/lib/kubelet/pods/d8170541-ea8d-400b-9636-52a81628985f/volumes" Nov 03 20:55:43.552000 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:43.551959 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz"] Nov 03 20:55:43.552962 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:43.552939 4493 cpu_manager.go:414] "containerMap: removing container" podUID="d8170541-ea8d-400b-9636-52a81628985f" containerName="istio-proxy" Nov 03 20:55:43.552962 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:43.552960 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8170541-ea8d-400b-9636-52a81628985f" containerName="istio-proxy" Nov 03 20:55:43.553155 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:43.552982 4493 cpu_manager.go:414] "containerMap: removing container" podUID="d8170541-ea8d-400b-9636-52a81628985f" containerName="istio-proxy" Nov 03 20:55:43.553155 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:43.552988 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8170541-ea8d-400b-9636-52a81628985f" containerName="istio-proxy" Nov 03 20:55:43.553155 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:43.553102 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8170541-ea8d-400b-9636-52a81628985f" containerName="istio-proxy" Nov 03 20:55:43.553155 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:43.553112 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8170541-ea8d-400b-9636-52a81628985f" containerName="istio-proxy" Nov 03 20:55:43.558962 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:43.558936 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz" Nov 03 20:55:43.570220 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:43.570190 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695ce88a-f841-4e61-b76a-d1a0f2b10136-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz\" (UID: \"695ce88a-f841-4e61-b76a-d1a0f2b10136\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz" Nov 03 20:55:43.570701 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:43.570467 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfrbl\" (UniqueName: \"kubernetes.io/projected/695ce88a-f841-4e61-b76a-d1a0f2b10136-kube-api-access-jfrbl\") pod \"payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz\" (UID: \"695ce88a-f841-4e61-b76a-d1a0f2b10136\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz" Nov 03 20:55:43.609083 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:43.609036 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz"] Nov 03 20:55:43.688358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:43.687940 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0e8374c4-dockercfg-rkwg9\"" Nov 03 20:55:43.688358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:43.688102 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0e8374c4\"" Nov 03 20:55:43.695644 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:43.690625 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfrbl\" (UniqueName: \"kubernetes.io/projected/695ce88a-f841-4e61-b76a-d1a0f2b10136-kube-api-access-jfrbl\") pod \"payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz\" (UID: \"695ce88a-f841-4e61-b76a-d1a0f2b10136\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz" Nov 03 20:55:43.695644 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:43.691022 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695ce88a-f841-4e61-b76a-d1a0f2b10136-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz\" (UID: \"695ce88a-f841-4e61-b76a-d1a0f2b10136\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz" Nov 03 20:55:43.695644 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:43.692438 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695ce88a-f841-4e61-b76a-d1a0f2b10136-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz\" (UID: \"695ce88a-f841-4e61-b76a-d1a0f2b10136\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz" Nov 03 20:55:43.754358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:43.752916 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfrbl\" (UniqueName: \"kubernetes.io/projected/695ce88a-f841-4e61-b76a-d1a0f2b10136-kube-api-access-jfrbl\") pod \"payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz\" (UID: \"695ce88a-f841-4e61-b76a-d1a0f2b10136\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz" Nov 03 20:55:43.876876 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:43.874818 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz" Nov 03 20:55:43.918407 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:43.908045 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc"] Nov 03 20:55:43.918407 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:43.908310 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc" podUID="baf717fb-3da6-4351-be0a-392cbc4434f2" containerName="maas-api" containerID="cri-o://5e2de5e0287eb2e72f3df7589b173d3d7875a1bfecd39f8395196bb9291723d6" gracePeriod=30 Nov 03 20:55:44.346165 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:44.346139 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc" Nov 03 20:55:44.398223 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:44.398192 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k"] Nov 03 20:55:44.451180 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:44.451090 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz"] Nov 03 20:55:44.462309 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:44.462277 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baf717fb-3da6-4351-be0a-392cbc4434f2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/baf717fb-3da6-4351-be0a-392cbc4434f2-metrics-certs\") pod \"baf717fb-3da6-4351-be0a-392cbc4434f2\" (UID: \"baf717fb-3da6-4351-be0a-392cbc4434f2\") " Nov 03 20:55:44.465232 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:44.463967 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baf717fb-3da6-4351-be0a-392cbc4434f2-kube-api-access-jg5rn\" (UniqueName: \"kubernetes.io/projected/baf717fb-3da6-4351-be0a-392cbc4434f2-kube-api-access-jg5rn\") pod \"baf717fb-3da6-4351-be0a-392cbc4434f2\" (UID: \"baf717fb-3da6-4351-be0a-392cbc4434f2\") " Nov 03 20:55:44.465232 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:44.464071 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baf717fb-3da6-4351-be0a-392cbc4434f2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf717fb-3da6-4351-be0a-392cbc4434f2-openshift-service-ca\") pod \"baf717fb-3da6-4351-be0a-392cbc4434f2\" (UID: \"baf717fb-3da6-4351-be0a-392cbc4434f2\") " Nov 03 20:55:44.465232 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:44.464387 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baf717fb-3da6-4351-be0a-392cbc4434f2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baf717fb-3da6-4351-be0a-392cbc4434f2-maas-api-tls\") pod \"baf717fb-3da6-4351-be0a-392cbc4434f2\" (UID: \"baf717fb-3da6-4351-be0a-392cbc4434f2\") " Nov 03 20:55:44.465232 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:44.464518 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf717fb-3da6-4351-be0a-392cbc4434f2-openshift-service-ca" pod "baf717fb-3da6-4351-be0a-392cbc4434f2" (UID: "baf717fb-3da6-4351-be0a-392cbc4434f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:44.465232 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:44.465200 4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf717fb-3da6-4351-be0a-392cbc4434f2-openshift-service-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:44.468223 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:44.467972 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baf717fb-3da6-4351-be0a-392cbc4434f2-maas-api-tls" pod "baf717fb-3da6-4351-be0a-392cbc4434f2" (UID: "baf717fb-3da6-4351-be0a-392cbc4434f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:55:44.468223 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:44.467999 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf717fb-3da6-4351-be0a-392cbc4434f2-kube-api-access-jg5rn" pod "baf717fb-3da6-4351-be0a-392cbc4434f2" (UID: "baf717fb-3da6-4351-be0a-392cbc4434f2"). InnerVolumeSpecName "kube-api-access-jg5rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:44.469725 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:44.469682 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baf717fb-3da6-4351-be0a-392cbc4434f2-metrics-certs" pod "baf717fb-3da6-4351-be0a-392cbc4434f2" (UID: "baf717fb-3da6-4351-be0a-392cbc4434f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:55:44.571064 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:44.571034 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baf717fb-3da6-4351-be0a-392cbc4434f2-maas-api-tls\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:44.571476 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:44.571070 4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baf717fb-3da6-4351-be0a-392cbc4434f2-metrics-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:44.571476 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:44.571089 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg5rn\" (UniqueName: \"kubernetes.io/projected/baf717fb-3da6-4351-be0a-392cbc4434f2-kube-api-access-jg5rn\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:44.812496 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:44.812455 4493 generic.go:356] "Generic (PLEG): container finished" podID="baf717fb-3da6-4351-be0a-392cbc4434f2" containerID="5e2de5e0287eb2e72f3df7589b173d3d7875a1bfecd39f8395196bb9291723d6" exitCode=0 Nov 03 20:55:44.812685 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:44.812512 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc" Nov 03 20:55:44.812685 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:44.812540 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc" event={"ID":"baf717fb-3da6-4351-be0a-392cbc4434f2","Type":"ContainerDied","Data":"5e2de5e0287eb2e72f3df7589b173d3d7875a1bfecd39f8395196bb9291723d6"} Nov 03 20:55:44.812685 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:44.812597 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc" event={"ID":"baf717fb-3da6-4351-be0a-392cbc4434f2","Type":"ContainerDied","Data":"effa9e4ae0dc94bf0ecc11380e065a3939a5aa53d3ea2c5cc4f9f19df87471e5"} Nov 03 20:55:44.812685 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:44.812628 4493 scope.go:122] "RemoveContainer" containerID="5e2de5e0287eb2e72f3df7589b173d3d7875a1bfecd39f8395196bb9291723d6" Nov 03 20:55:44.815539 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:44.815508 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz" event={"ID":"695ce88a-f841-4e61-b76a-d1a0f2b10136","Type":"ContainerStarted","Data":"74fabb929a7ebcc51c3be1588e34041d4c56b1ce883adf7107b6fe70bfefb497"} Nov 03 20:55:44.815748 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:44.815724 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz" event={"ID":"695ce88a-f841-4e61-b76a-d1a0f2b10136","Type":"ContainerStarted","Data":"b991c5594492dcbc4cbc9d642dd6aa9c106c8d1066342925fa17c17e13a89660"} Nov 03 20:55:44.815867 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:44.815759 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz" Nov 03 20:55:44.824300 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:44.824285 4493 scope.go:122] "RemoveContainer" containerID="5e2de5e0287eb2e72f3df7589b173d3d7875a1bfecd39f8395196bb9291723d6" Nov 03 20:55:44.824635 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:55:44.824611 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e2de5e0287eb2e72f3df7589b173d3d7875a1bfecd39f8395196bb9291723d6\": container with ID starting with 5e2de5e0287eb2e72f3df7589b173d3d7875a1bfecd39f8395196bb9291723d6 not found: ID does not exist" containerID="5e2de5e0287eb2e72f3df7589b173d3d7875a1bfecd39f8395196bb9291723d6" Nov 03 20:55:44.824699 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:44.824644 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e2de5e0287eb2e72f3df7589b173d3d7875a1bfecd39f8395196bb9291723d6"} err="failed to get container status \"5e2de5e0287eb2e72f3df7589b173d3d7875a1bfecd39f8395196bb9291723d6\": rpc error: code = NotFound desc = could not find container \"5e2de5e0287eb2e72f3df7589b173d3d7875a1bfecd39f8395196bb9291723d6\": container with ID starting with 5e2de5e0287eb2e72f3df7589b173d3d7875a1bfecd39f8395196bb9291723d6 not found: ID does not exist" Nov 03 20:55:44.982380 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:44.982302 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz" podStartSLOduration=1.9822856679999998 podStartE2EDuration="1.982285668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:44.977236074 +0000 UTC m=+1616.535226576" watchObservedRunningTime="2026-11-03 20:55:44.982285668 +0000 UTC m=+1616.540276167" Nov 03 20:55:45.085318 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:45.085262 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc"] Nov 03 20:55:45.090940 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:45.090913 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-43693d42-6786657c56-khlxc"] Nov 03 20:55:46.241626 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:46.241585 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4"] Nov 03 20:55:46.242825 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:46.242792 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" podUID="283bf8dd-6d9c-46b5-9bc1-3e88567d3af1" containerName="istio-proxy" containerID="cri-o://ad893a1354c8507bfec4ff6d9dec8734958079da8435b2a8d82a8d8d483d1685" gracePeriod=30 Nov 03 20:55:46.570236 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:46.570200 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baf717fb-3da6-4351-be0a-392cbc4434f2" path="/var/lib/kubelet/pods/baf717fb-3da6-4351-be0a-392cbc4434f2/volumes" Nov 03 20:55:47.304443 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.293004 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4"] Nov 03 20:55:47.304443 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.294314 4493 cpu_manager.go:414] "containerMap: removing container" podUID="baf717fb-3da6-4351-be0a-392cbc4434f2" containerName="maas-api" Nov 03 20:55:47.304443 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.294356 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baf717fb-3da6-4351-be0a-392cbc4434f2" containerName="maas-api" Nov 03 20:55:47.304443 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.294597 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="baf717fb-3da6-4351-be0a-392cbc4434f2" containerName="maas-api" Nov 03 20:55:47.304443 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.301156 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.352234 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.352183 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4"] Nov 03 20:55:47.394407 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.382365 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istiod-ca-cert\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.394407 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.382474 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-data\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.394407 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.382526 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-credential-socket\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.394407 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.382577 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-workload-socket\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.394407 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.382680 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-workload-certs\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.394407 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.382713 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-envoy\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.394407 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.382751 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-podinfo\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.394407 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.382820 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94sq6\" (UniqueName: \"kubernetes.io/projected/0d3000f1-172d-4b7b-9d45-fa04ef63145f-kube-api-access-94sq6\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.394407 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.382973 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-token\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.394407 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.392172 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-cc15b1d7-openshift-default-dockercfg-mjnvv\"" Nov 03 20:55:47.486674 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.485523 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istiod-ca-cert\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.486674 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.485636 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-data\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.486674 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.485698 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-credential-socket\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.486674 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.485748 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-workload-socket\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.486674 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.485853 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-workload-certs\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.486674 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.485888 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-envoy\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.486674 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.485923 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-podinfo\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.486674 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.485994 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94sq6\" (UniqueName: \"kubernetes.io/projected/0d3000f1-172d-4b7b-9d45-fa04ef63145f-kube-api-access-94sq6\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.486674 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.486143 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-token\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.489471 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.488159 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istiod-ca-cert\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.489471 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.488445 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-data\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.489471 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.488663 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-credential-socket\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.489471 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.488867 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-workload-socket\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.489471 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.489068 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-workload-certs\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.498447 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.496663 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-envoy\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.503479 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.503412 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-podinfo\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.544000 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.543963 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-token\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.546298 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.545443 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94sq6\" (UniqueName: \"kubernetes.io/projected/0d3000f1-172d-4b7b-9d45-fa04ef63145f-kube-api-access-94sq6\") pod \"e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:47.693402 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:47.693312 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:48.166972 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:48.166941 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4"] Nov 03 20:55:48.210972 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:55:48.210938 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3000f1_172d_4b7b_9d45_fa04ef63145f.slice/crio-abf671b0bc7c7aa34a6b013de37db3fc2f4d1b670973d1650b1a5b8b5293ad2b WatchSource:0}: Error finding container abf671b0bc7c7aa34a6b013de37db3fc2f4d1b670973d1650b1a5b8b5293ad2b: Status 404 returned error can't find the container with id abf671b0bc7c7aa34a6b013de37db3fc2f4d1b670973d1650b1a5b8b5293ad2b Nov 03 20:55:48.214077 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:48.214045 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:55:48.214155 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:48.214118 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:55:48.214155 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:48.214147 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:55:48.861646 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:48.861609 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" event={"ID":"0d3000f1-172d-4b7b-9d45-fa04ef63145f","Type":"ContainerStarted","Data":"ec76955c5127e08a3ce1b7f2d997987b8d7360848ce06ce1a81ce2fcb71cc88a"} Nov 03 20:55:48.862126 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:48.861696 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" event={"ID":"0d3000f1-172d-4b7b-9d45-fa04ef63145f","Type":"ContainerStarted","Data":"abf671b0bc7c7aa34a6b013de37db3fc2f4d1b670973d1650b1a5b8b5293ad2b"} Nov 03 20:55:49.023586 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:49.023534 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" podStartSLOduration=2.023519663 podStartE2EDuration="2.023519663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:49.018518737 +0000 UTC m=+1620.576509267" watchObservedRunningTime="2026-11-03 20:55:49.023519663 +0000 UTC m=+1620.581510239" Nov 03 20:55:49.398962 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:49.398904 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k" podUID="333cb4b6-4674-462d-a66d-a00198468cb7" containerName="payload-pre-processing" containerID="cri-o://cc07146b9913000cea16d6fc2bcfbb1b6cda93a6948b652b20747ce2d2e55f67" gracePeriod=25 Nov 03 20:55:49.692184 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:49.692152 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k" Nov 03 20:55:49.693948 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:49.693922 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:49.700496 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:49.700471 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:49.826067 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:49.826031 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/333cb4b6-4674-462d-a66d-a00198468cb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/333cb4b6-4674-462d-a66d-a00198468cb7-plugins-config-volume\") pod \"333cb4b6-4674-462d-a66d-a00198468cb7\" (UID: \"333cb4b6-4674-462d-a66d-a00198468cb7\") " Nov 03 20:55:49.826213 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:49.826155 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/333cb4b6-4674-462d-a66d-a00198468cb7-kube-api-access-rjxjm\" (UniqueName: \"kubernetes.io/projected/333cb4b6-4674-462d-a66d-a00198468cb7-kube-api-access-rjxjm\") pod \"333cb4b6-4674-462d-a66d-a00198468cb7\" (UID: \"333cb4b6-4674-462d-a66d-a00198468cb7\") " Nov 03 20:55:49.826514 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:49.826472 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/333cb4b6-4674-462d-a66d-a00198468cb7-plugins-config-volume" pod "333cb4b6-4674-462d-a66d-a00198468cb7" (UID: "333cb4b6-4674-462d-a66d-a00198468cb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:49.827114 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:49.827090 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/333cb4b6-4674-462d-a66d-a00198468cb7-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:49.828615 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:49.828588 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/333cb4b6-4674-462d-a66d-a00198468cb7-kube-api-access-rjxjm" pod "333cb4b6-4674-462d-a66d-a00198468cb7" (UID: "333cb4b6-4674-462d-a66d-a00198468cb7"). InnerVolumeSpecName "kube-api-access-rjxjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:49.870573 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:49.870544 4493 generic.go:356] "Generic (PLEG): container finished" podID="333cb4b6-4674-462d-a66d-a00198468cb7" containerID="cc07146b9913000cea16d6fc2bcfbb1b6cda93a6948b652b20747ce2d2e55f67" exitCode=0 Nov 03 20:55:49.870963 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:49.870588 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k" Nov 03 20:55:49.870963 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:49.870630 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k" event={"ID":"333cb4b6-4674-462d-a66d-a00198468cb7","Type":"ContainerDied","Data":"cc07146b9913000cea16d6fc2bcfbb1b6cda93a6948b652b20747ce2d2e55f67"} Nov 03 20:55:49.870963 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:49.870680 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k" event={"ID":"333cb4b6-4674-462d-a66d-a00198468cb7","Type":"ContainerDied","Data":"0c9e198317a24d1bcf58b44b1dbefc0cbd1ae9efad9d1368653f9b212819730d"} Nov 03 20:55:49.870963 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:49.870703 4493 scope.go:122] "RemoveContainer" containerID="cc07146b9913000cea16d6fc2bcfbb1b6cda93a6948b652b20747ce2d2e55f67" Nov 03 20:55:49.871165 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:49.871032 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:49.872234 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:49.872215 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:55:49.884886 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:49.884857 4493 scope.go:122] "RemoveContainer" containerID="cc07146b9913000cea16d6fc2bcfbb1b6cda93a6948b652b20747ce2d2e55f67" Nov 03 20:55:49.885202 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:55:49.885181 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc07146b9913000cea16d6fc2bcfbb1b6cda93a6948b652b20747ce2d2e55f67\": container with ID starting with cc07146b9913000cea16d6fc2bcfbb1b6cda93a6948b652b20747ce2d2e55f67 not found: ID does not exist" containerID="cc07146b9913000cea16d6fc2bcfbb1b6cda93a6948b652b20747ce2d2e55f67" Nov 03 20:55:49.885264 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:49.885211 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc07146b9913000cea16d6fc2bcfbb1b6cda93a6948b652b20747ce2d2e55f67"} err="failed to get container status \"cc07146b9913000cea16d6fc2bcfbb1b6cda93a6948b652b20747ce2d2e55f67\": rpc error: code = NotFound desc = could not find container \"cc07146b9913000cea16d6fc2bcfbb1b6cda93a6948b652b20747ce2d2e55f67\": container with ID starting with cc07146b9913000cea16d6fc2bcfbb1b6cda93a6948b652b20747ce2d2e55f67 not found: ID does not exist" Nov 03 20:55:49.928915 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:49.928856 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjxjm\" (UniqueName: \"kubernetes.io/projected/333cb4b6-4674-462d-a66d-a00198468cb7-kube-api-access-rjxjm\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:50.142100 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:50.142056 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k"] Nov 03 20:55:50.146501 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:50.146477 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e3e5cd62-646fd4799-hq76k"] Nov 03 20:55:50.563773 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:50.563743 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="333cb4b6-4674-462d-a66d-a00198468cb7" path="/var/lib/kubelet/pods/333cb4b6-4674-462d-a66d-a00198468cb7/volumes" Nov 03 20:55:51.589661 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.589639 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:55:51.662322 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.662295 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istiod-ca-cert\") pod \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " Nov 03 20:55:51.662501 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.662392 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-credential-socket\") pod \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " Nov 03 20:55:51.662562 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.662495 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-kube-api-access-lxmjs\" (UniqueName: \"kubernetes.io/projected/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-kube-api-access-lxmjs\") pod \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " Nov 03 20:55:51.662606 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.662557 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-data\") pod \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " Nov 03 20:55:51.662719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.662696 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istiod-ca-cert" pod "283bf8dd-6d9c-46b5-9bc1-3e88567d3af1" (UID: "283bf8dd-6d9c-46b5-9bc1-3e88567d3af1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:51.662719 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.662708 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-workload-certs\") pod \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " Nov 03 20:55:51.662902 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.662726 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-credential-socket" pod "283bf8dd-6d9c-46b5-9bc1-3e88567d3af1" (UID: "283bf8dd-6d9c-46b5-9bc1-3e88567d3af1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:51.662902 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.662760 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-envoy\") pod \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " Nov 03 20:55:51.662902 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.662793 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-podinfo\") pod \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " Nov 03 20:55:51.662902 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.662838 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-workload-socket\") pod \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " Nov 03 20:55:51.662902 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.662851 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-data" pod "283bf8dd-6d9c-46b5-9bc1-3e88567d3af1" (UID: "283bf8dd-6d9c-46b5-9bc1-3e88567d3af1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:51.663171 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.662929 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-token\" (UniqueName: \"kubernetes.io/projected/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-token\") pod \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\" (UID: \"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1\") " Nov 03 20:55:51.663171 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.662953 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-workload-certs" pod "283bf8dd-6d9c-46b5-9bc1-3e88567d3af1" (UID: "283bf8dd-6d9c-46b5-9bc1-3e88567d3af1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:51.663171 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.663138 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-workload-socket" pod "283bf8dd-6d9c-46b5-9bc1-3e88567d3af1" (UID: "283bf8dd-6d9c-46b5-9bc1-3e88567d3af1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:51.663633 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.663614 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:51.663714 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.663638 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:51.663714 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.663652 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:51.663714 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.663672 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:51.663714 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.663682 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:51.664980 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.664954 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-envoy" pod "283bf8dd-6d9c-46b5-9bc1-3e88567d3af1" (UID: "283bf8dd-6d9c-46b5-9bc1-3e88567d3af1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:51.665075 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.665057 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-kube-api-access-lxmjs" pod "283bf8dd-6d9c-46b5-9bc1-3e88567d3af1" (UID: "283bf8dd-6d9c-46b5-9bc1-3e88567d3af1"). InnerVolumeSpecName "kube-api-access-lxmjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:51.665558 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.665534 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-token" pod "283bf8dd-6d9c-46b5-9bc1-3e88567d3af1" (UID: "283bf8dd-6d9c-46b5-9bc1-3e88567d3af1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:51.665558 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.665535 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-podinfo" pod "283bf8dd-6d9c-46b5-9bc1-3e88567d3af1" (UID: "283bf8dd-6d9c-46b5-9bc1-3e88567d3af1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:55:51.746713 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:55:51.746674 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:55:51.765662 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.765629 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:51.765662 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.765664 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:51.765845 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.765681 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:51.765845 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.765694 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxmjs\" (UniqueName: \"kubernetes.io/projected/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1-kube-api-access-lxmjs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:51.887820 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.887742 4493 generic.go:356] "Generic (PLEG): container finished" podID="283bf8dd-6d9c-46b5-9bc1-3e88567d3af1" containerID="ad893a1354c8507bfec4ff6d9dec8734958079da8435b2a8d82a8d8d483d1685" exitCode=0 Nov 03 20:55:51.887820 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.887789 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" Nov 03 20:55:51.888022 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.887820 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" event={"ID":"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1","Type":"ContainerDied","Data":"ad893a1354c8507bfec4ff6d9dec8734958079da8435b2a8d82a8d8d483d1685"} Nov 03 20:55:51.888022 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.887876 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4" event={"ID":"283bf8dd-6d9c-46b5-9bc1-3e88567d3af1","Type":"ContainerDied","Data":"3fd622843c423bd469dc3a9bb773a311ba0a416a5e7daf128df674603cd51029"} Nov 03 20:55:51.888022 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.887903 4493 scope.go:122] "RemoveContainer" containerID="ad893a1354c8507bfec4ff6d9dec8734958079da8435b2a8d82a8d8d483d1685" Nov 03 20:55:51.899516 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.899500 4493 scope.go:122] "RemoveContainer" containerID="0b55a24f7062046592a97b6e3d19b652f78dc88847b1bb4386d02dcf6eee2c24" Nov 03 20:55:51.908929 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.908912 4493 scope.go:122] "RemoveContainer" containerID="ad893a1354c8507bfec4ff6d9dec8734958079da8435b2a8d82a8d8d483d1685" Nov 03 20:55:51.909178 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:55:51.909162 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad893a1354c8507bfec4ff6d9dec8734958079da8435b2a8d82a8d8d483d1685\": container with ID starting with ad893a1354c8507bfec4ff6d9dec8734958079da8435b2a8d82a8d8d483d1685 not found: ID does not exist" containerID="ad893a1354c8507bfec4ff6d9dec8734958079da8435b2a8d82a8d8d483d1685" Nov 03 20:55:51.909231 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.909187 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad893a1354c8507bfec4ff6d9dec8734958079da8435b2a8d82a8d8d483d1685"} err="failed to get container status \"ad893a1354c8507bfec4ff6d9dec8734958079da8435b2a8d82a8d8d483d1685\": rpc error: code = NotFound desc = could not find container \"ad893a1354c8507bfec4ff6d9dec8734958079da8435b2a8d82a8d8d483d1685\": container with ID starting with ad893a1354c8507bfec4ff6d9dec8734958079da8435b2a8d82a8d8d483d1685 not found: ID does not exist" Nov 03 20:55:51.909231 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.909204 4493 scope.go:122] "RemoveContainer" containerID="0b55a24f7062046592a97b6e3d19b652f78dc88847b1bb4386d02dcf6eee2c24" Nov 03 20:55:51.909471 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:55:51.909452 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b55a24f7062046592a97b6e3d19b652f78dc88847b1bb4386d02dcf6eee2c24\": container with ID starting with 0b55a24f7062046592a97b6e3d19b652f78dc88847b1bb4386d02dcf6eee2c24 not found: ID does not exist" containerID="0b55a24f7062046592a97b6e3d19b652f78dc88847b1bb4386d02dcf6eee2c24" Nov 03 20:55:51.909543 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:51.909479 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b55a24f7062046592a97b6e3d19b652f78dc88847b1bb4386d02dcf6eee2c24"} err="failed to get container status \"0b55a24f7062046592a97b6e3d19b652f78dc88847b1bb4386d02dcf6eee2c24\": rpc error: code = NotFound desc = could not find container \"0b55a24f7062046592a97b6e3d19b652f78dc88847b1bb4386d02dcf6eee2c24\": container with ID starting with 0b55a24f7062046592a97b6e3d19b652f78dc88847b1bb4386d02dcf6eee2c24 not found: ID does not exist" Nov 03 20:55:52.046690 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:52.046640 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4"] Nov 03 20:55:52.054518 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:52.054482 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-43693d42-openshift-default-5d5dd9575f-p6pc4"] Nov 03 20:55:52.563643 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:52.563607 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="283bf8dd-6d9c-46b5-9bc1-3e88567d3af1" path="/var/lib/kubelet/pods/283bf8dd-6d9c-46b5-9bc1-3e88567d3af1/volumes" Nov 03 20:55:54.686356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:54.682410 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz"] Nov 03 20:55:54.686356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:54.683492 4493 cpu_manager.go:414] "containerMap: removing container" podUID="333cb4b6-4674-462d-a66d-a00198468cb7" containerName="payload-pre-processing" Nov 03 20:55:54.686356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:54.683512 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="333cb4b6-4674-462d-a66d-a00198468cb7" containerName="payload-pre-processing" Nov 03 20:55:54.686356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:54.683605 4493 cpu_manager.go:414] "containerMap: removing container" podUID="283bf8dd-6d9c-46b5-9bc1-3e88567d3af1" containerName="istio-proxy" Nov 03 20:55:54.686356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:54.683614 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="283bf8dd-6d9c-46b5-9bc1-3e88567d3af1" containerName="istio-proxy" Nov 03 20:55:54.686356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:54.683638 4493 cpu_manager.go:414] "containerMap: removing container" podUID="283bf8dd-6d9c-46b5-9bc1-3e88567d3af1" containerName="istio-proxy" Nov 03 20:55:54.686356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:54.683645 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="283bf8dd-6d9c-46b5-9bc1-3e88567d3af1" containerName="istio-proxy" Nov 03 20:55:54.686356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:54.683817 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="283bf8dd-6d9c-46b5-9bc1-3e88567d3af1" containerName="istio-proxy" Nov 03 20:55:54.686356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:54.683835 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="283bf8dd-6d9c-46b5-9bc1-3e88567d3af1" containerName="istio-proxy" Nov 03 20:55:54.686356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:54.683844 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="333cb4b6-4674-462d-a66d-a00198468cb7" containerName="payload-pre-processing" Nov 03 20:55:54.691467 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:54.691440 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz" Nov 03 20:55:54.733356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:54.719794 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d59bd1c6-7591-4470-b328-b0d60fb6f292-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz\" (UID: \"d59bd1c6-7591-4470-b328-b0d60fb6f292\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz" Nov 03 20:55:54.733356 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:54.720083 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtvm2\" (UniqueName: \"kubernetes.io/projected/d59bd1c6-7591-4470-b328-b0d60fb6f292-kube-api-access-vtvm2\") pod \"payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz\" (UID: \"d59bd1c6-7591-4470-b328-b0d60fb6f292\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz" Nov 03 20:55:54.760054 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:54.753992 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz"] Nov 03 20:55:54.796492 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:54.796460 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-cc15b1d7\"" Nov 03 20:55:54.798218 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:54.797511 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-cc15b1d7-dockercfg-nlvzf\"" Nov 03 20:55:54.823114 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:54.821285 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d59bd1c6-7591-4470-b328-b0d60fb6f292-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz\" (UID: \"d59bd1c6-7591-4470-b328-b0d60fb6f292\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz" Nov 03 20:55:54.823114 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:54.821666 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtvm2\" (UniqueName: \"kubernetes.io/projected/d59bd1c6-7591-4470-b328-b0d60fb6f292-kube-api-access-vtvm2\") pod \"payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz\" (UID: \"d59bd1c6-7591-4470-b328-b0d60fb6f292\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz" Nov 03 20:55:54.823114 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:54.823044 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d59bd1c6-7591-4470-b328-b0d60fb6f292-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz\" (UID: \"d59bd1c6-7591-4470-b328-b0d60fb6f292\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz" Nov 03 20:55:54.828917 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:54.828892 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8"] Nov 03 20:55:54.834269 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:54.834244 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8" Nov 03 20:55:54.958488 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:54.940650 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ae7b36-9635-4ea7-badb-a5a97321c841-plugins-config-volume\") pod \"payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8\" (UID: \"f0ae7b36-9635-4ea7-badb-a5a97321c841\") " pod="openshift-ingress/payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8" Nov 03 20:55:54.958488 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:54.940991 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5jhw\" (UniqueName: \"kubernetes.io/projected/f0ae7b36-9635-4ea7-badb-a5a97321c841-kube-api-access-l5jhw\") pod \"payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8\" (UID: \"f0ae7b36-9635-4ea7-badb-a5a97321c841\") " pod="openshift-ingress/payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8" Nov 03 20:55:54.958488 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:54.952716 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtvm2\" (UniqueName: \"kubernetes.io/projected/d59bd1c6-7591-4470-b328-b0d60fb6f292-kube-api-access-vtvm2\") pod \"payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz\" (UID: \"d59bd1c6-7591-4470-b328-b0d60fb6f292\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz" Nov 03 20:55:54.958488 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:54.954921 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8"] Nov 03 20:55:55.046382 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:55.044417 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ae7b36-9635-4ea7-badb-a5a97321c841-plugins-config-volume\") pod \"payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8\" (UID: \"f0ae7b36-9635-4ea7-badb-a5a97321c841\") " pod="openshift-ingress/payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8" Nov 03 20:55:55.046382 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:55.044625 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5jhw\" (UniqueName: \"kubernetes.io/projected/f0ae7b36-9635-4ea7-badb-a5a97321c841-kube-api-access-l5jhw\") pod \"payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8\" (UID: \"f0ae7b36-9635-4ea7-badb-a5a97321c841\") " pod="openshift-ingress/payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8" Nov 03 20:55:55.046382 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:55.046216 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ae7b36-9635-4ea7-badb-a5a97321c841-plugins-config-volume\") pod \"payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8\" (UID: \"f0ae7b36-9635-4ea7-badb-a5a97321c841\") " pod="openshift-ingress/payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8" Nov 03 20:55:55.080075 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:55.080034 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz" Nov 03 20:55:55.099917 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:55.099882 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5jhw\" (UniqueName: \"kubernetes.io/projected/f0ae7b36-9635-4ea7-badb-a5a97321c841-kube-api-access-l5jhw\") pod \"payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8\" (UID: \"f0ae7b36-9635-4ea7-badb-a5a97321c841\") " pod="openshift-ingress/payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8" Nov 03 20:55:55.235603 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:55.233766 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8" Nov 03 20:55:55.557431 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:55.545828 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz"] Nov 03 20:55:55.662090 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:55.662051 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8"] Nov 03 20:55:55.827633 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:55.827559 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz" Nov 03 20:55:55.975625 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:55.975589 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8" event={"ID":"f0ae7b36-9635-4ea7-badb-a5a97321c841","Type":"ContainerStarted","Data":"2934f102559eb5d5832546fe5b39fdb7dbe80335dfa4281fe9dac4d845da9816"} Nov 03 20:55:55.975794 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:55.975661 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8" event={"ID":"f0ae7b36-9635-4ea7-badb-a5a97321c841","Type":"ContainerStarted","Data":"6f7bf6f9aaa17ce036880ef4c739fde1de7199d8bcaf1730ac944b2bca422987"} Nov 03 20:55:55.975794 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:55.975688 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8" Nov 03 20:55:55.977577 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:55.977544 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz" event={"ID":"d59bd1c6-7591-4470-b328-b0d60fb6f292","Type":"ContainerStarted","Data":"57a6a75431b24b9a0de9ca3ae3cece7137ae8a0a99dd3efe08d9746a9bb9e8d9"} Nov 03 20:55:55.977696 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:55.977598 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz" event={"ID":"d59bd1c6-7591-4470-b328-b0d60fb6f292","Type":"ContainerStarted","Data":"c2d0225c5b47d0e91c4203588e51ab67ec13f3001bc6415a1eae227d507573bd"} Nov 03 20:55:55.977696 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:55.977644 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz" Nov 03 20:55:56.085827 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:56.085720 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8" podStartSLOduration=2.085701876 podStartE2EDuration="2.085701876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:56.081983134 +0000 UTC m=+1627.639973634" watchObservedRunningTime="2026-11-03 20:55:56.085701876 +0000 UTC m=+1627.643692378" Nov 03 20:55:56.190801 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:55:56.190749 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz" podStartSLOduration=2.190735587 podStartE2EDuration="2.190735587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:56.188375944 +0000 UTC m=+1627.746366515" watchObservedRunningTime="2026-11-03 20:55:56.190735587 +0000 UTC m=+1627.748726085" Nov 03 20:56:01.495760 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.495724 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn"] Nov 03 20:56:01.500138 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.500115 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 20:56:01.549383 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.549350 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 20:56:01.549383 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.549355 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 20:56:01.549638 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.549437 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 20:56:01.584693 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.584656 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7028d8a7-879e-4768-8bf2-390f98c23ac8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g4xjn\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 20:56:01.584869 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.584702 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj2rh\" (UniqueName: \"kubernetes.io/projected/7028d8a7-879e-4768-8bf2-390f98c23ac8-kube-api-access-xj2rh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g4xjn\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 20:56:01.584966 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.584938 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7028d8a7-879e-4768-8bf2-390f98c23ac8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g4xjn\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 20:56:01.585038 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.585013 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g4xjn\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 20:56:01.585096 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.585066 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g4xjn\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 20:56:01.585254 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.585230 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g4xjn\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 20:56:01.585434 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.585403 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g4xjn\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 20:56:01.597217 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.596503 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4mvmz\"" Nov 03 20:56:01.603992 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.603957 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn"] Nov 03 20:56:01.686497 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.686456 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g4xjn\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 20:56:01.686694 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.686598 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g4xjn\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 20:56:01.687060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.687034 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7028d8a7-879e-4768-8bf2-390f98c23ac8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g4xjn\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 20:56:01.687162 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.687105 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj2rh\" (UniqueName: \"kubernetes.io/projected/7028d8a7-879e-4768-8bf2-390f98c23ac8-kube-api-access-xj2rh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g4xjn\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 20:56:01.687292 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.687270 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g4xjn\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 20:56:01.687292 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.687286 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7028d8a7-879e-4768-8bf2-390f98c23ac8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g4xjn\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 20:56:01.687478 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.687373 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g4xjn\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 20:56:01.687478 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.687416 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g4xjn\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 20:56:01.690237 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.690209 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7028d8a7-879e-4768-8bf2-390f98c23ac8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g4xjn\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 20:56:01.690578 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.690557 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g4xjn\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 20:56:01.690578 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.690564 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g4xjn\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 20:56:01.699608 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.699581 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7028d8a7-879e-4768-8bf2-390f98c23ac8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g4xjn\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 20:56:01.754209 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.754139 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj2rh\" (UniqueName: \"kubernetes.io/projected/7028d8a7-879e-4768-8bf2-390f98c23ac8-kube-api-access-xj2rh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g4xjn\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 20:56:01.754746 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.754725 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g4xjn\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 20:56:01.816259 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:01.816230 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 20:56:02.313512 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:02.313487 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn"] Nov 03 20:56:02.315799 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:56:02.315771 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7028d8a7_879e_4768_8bf2_390f98c23ac8.slice/crio-a06500fd3ebff2b4b22a525e4a23c798d0f2d8aa9350135b1b64f3a6d119a74f WatchSource:0}: Error finding container a06500fd3ebff2b4b22a525e4a23c798d0f2d8aa9350135b1b64f3a6d119a74f: Status 404 returned error can't find the container with id a06500fd3ebff2b4b22a525e4a23c798d0f2d8aa9350135b1b64f3a6d119a74f Nov 03 20:56:03.079378 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:03.079325 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" event={"ID":"7028d8a7-879e-4768-8bf2-390f98c23ac8","Type":"ContainerStarted","Data":"a06500fd3ebff2b4b22a525e4a23c798d0f2d8aa9350135b1b64f3a6d119a74f"} Nov 03 20:56:03.345905 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:03.345820 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7"] Nov 03 20:56:03.346055 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:03.346030 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" podUID="1f96068d-fb57-4a30-876a-e3158d210c3c" containerName="istio-proxy" containerID="cri-o://b393a9b8a28d83ec6735dc5a54e18f7073e4a9f406519cf338a769ca670c8f7c" gracePeriod=30 Nov 03 20:56:06.360972 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:06.360943 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:56:06.361266 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:06.361010 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:56:06.988925 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:06.988897 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz" Nov 03 20:56:06.989369 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:06.989324 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8" Nov 03 20:56:07.114701 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:07.114663 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" event={"ID":"7028d8a7-879e-4768-8bf2-390f98c23ac8","Type":"ContainerStarted","Data":"d36ad86441a16fd2a0ca3e640d240b21a9719266b27eeff7c814e141ebd206c3"} Nov 03 20:56:07.114879 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:07.114862 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 20:56:07.116721 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:07.116694 4493 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-g4xjn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:56:07.116854 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:07.116735 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" podUID="7028d8a7-879e-4768-8bf2-390f98c23ac8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:56:07.334998 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:07.334948 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" podStartSLOduration=2.293259491 podStartE2EDuration="6.334933392s" totalImagesPullingTime="4.041673901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:07.331569553 +0000 UTC m=+1638.889560075" watchObservedRunningTime="2026-11-03 20:56:07.334933392 +0000 UTC m=+1638.892923890" Nov 03 20:56:07.403234 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:07.403198 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw"] Nov 03 20:56:07.403734 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:07.403468 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw" podUID="059ef4a2-73a5-4b8e-9446-fe080a5bb3d7" containerName="maas-api" containerID="cri-o://ab1a9d65db830f1957ef9ca1eb6943355a428adbc6775f37cf68e289194d3362" gracePeriod=30 Nov 03 20:56:07.715462 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:07.715428 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw" Nov 03 20:56:07.739626 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:07.739598 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-openshift-service-ca\") pod \"059ef4a2-73a5-4b8e-9446-fe080a5bb3d7\" (UID: \"059ef4a2-73a5-4b8e-9446-fe080a5bb3d7\") " Nov 03 20:56:07.739795 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:07.739782 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-kube-api-access-d6rnf\" (UniqueName: \"kubernetes.io/projected/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-kube-api-access-d6rnf\") pod \"059ef4a2-73a5-4b8e-9446-fe080a5bb3d7\" (UID: \"059ef4a2-73a5-4b8e-9446-fe080a5bb3d7\") " Nov 03 20:56:07.739859 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:07.739823 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-metrics-certs\") pod \"059ef4a2-73a5-4b8e-9446-fe080a5bb3d7\" (UID: \"059ef4a2-73a5-4b8e-9446-fe080a5bb3d7\") " Nov 03 20:56:07.739905 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:07.739884 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-maas-api-tls\") pod \"059ef4a2-73a5-4b8e-9446-fe080a5bb3d7\" (UID: \"059ef4a2-73a5-4b8e-9446-fe080a5bb3d7\") " Nov 03 20:56:07.740402 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:07.740326 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-openshift-service-ca" pod "059ef4a2-73a5-4b8e-9446-fe080a5bb3d7" (UID: "059ef4a2-73a5-4b8e-9446-fe080a5bb3d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:56:07.741012 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:07.740987 4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-openshift-service-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:07.742371 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:07.742304 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-metrics-certs" pod "059ef4a2-73a5-4b8e-9446-fe080a5bb3d7" (UID: "059ef4a2-73a5-4b8e-9446-fe080a5bb3d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:56:07.742371 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:07.742363 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-maas-api-tls" pod "059ef4a2-73a5-4b8e-9446-fe080a5bb3d7" (UID: "059ef4a2-73a5-4b8e-9446-fe080a5bb3d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:56:07.742529 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:07.742488 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-kube-api-access-d6rnf" pod "059ef4a2-73a5-4b8e-9446-fe080a5bb3d7" (UID: "059ef4a2-73a5-4b8e-9446-fe080a5bb3d7"). InnerVolumeSpecName "kube-api-access-d6rnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:56:07.844355 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:07.843099 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6rnf\" (UniqueName: \"kubernetes.io/projected/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-kube-api-access-d6rnf\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:07.844355 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:07.843137 4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-metrics-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:07.844355 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:07.843154 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7-maas-api-tls\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:08.157431 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.154552 4493 generic.go:356] "Generic (PLEG): container finished" podID="059ef4a2-73a5-4b8e-9446-fe080a5bb3d7" containerID="ab1a9d65db830f1957ef9ca1eb6943355a428adbc6775f37cf68e289194d3362" exitCode=0 Nov 03 20:56:08.157431 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.156106 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw" Nov 03 20:56:08.168433 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.168391 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw" event={"ID":"059ef4a2-73a5-4b8e-9446-fe080a5bb3d7","Type":"ContainerDied","Data":"ab1a9d65db830f1957ef9ca1eb6943355a428adbc6775f37cf68e289194d3362"} Nov 03 20:56:08.168610 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.168473 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw" event={"ID":"059ef4a2-73a5-4b8e-9446-fe080a5bb3d7","Type":"ContainerDied","Data":"573e952275375968abeaefe70e28b2844c4d99ac28c2294ebee90a6e125b5f16"} Nov 03 20:56:08.168610 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.168503 4493 scope.go:122] "RemoveContainer" containerID="ab1a9d65db830f1957ef9ca1eb6943355a428adbc6775f37cf68e289194d3362" Nov 03 20:56:08.183388 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.180830 4493 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-g4xjn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:56:08.183388 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.180887 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" podUID="7028d8a7-879e-4768-8bf2-390f98c23ac8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:56:08.215563 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.211459 4493 scope.go:122] "RemoveContainer" containerID="ab1a9d65db830f1957ef9ca1eb6943355a428adbc6775f37cf68e289194d3362" Nov 03 20:56:08.220398 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:56:08.216931 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab1a9d65db830f1957ef9ca1eb6943355a428adbc6775f37cf68e289194d3362\": container with ID starting with ab1a9d65db830f1957ef9ca1eb6943355a428adbc6775f37cf68e289194d3362 not found: ID does not exist" containerID="ab1a9d65db830f1957ef9ca1eb6943355a428adbc6775f37cf68e289194d3362" Nov 03 20:56:08.220398 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.216973 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab1a9d65db830f1957ef9ca1eb6943355a428adbc6775f37cf68e289194d3362"} err="failed to get container status \"ab1a9d65db830f1957ef9ca1eb6943355a428adbc6775f37cf68e289194d3362\": rpc error: code = NotFound desc = could not find container \"ab1a9d65db830f1957ef9ca1eb6943355a428adbc6775f37cf68e289194d3362\": container with ID starting with ab1a9d65db830f1957ef9ca1eb6943355a428adbc6775f37cf68e289194d3362 not found: ID does not exist" Nov 03 20:56:08.409413 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.407427 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw"] Nov 03 20:56:08.412923 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.412066 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e3e5cd62-865c4d999-44hgw"] Nov 03 20:56:08.663261 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.651789 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="059ef4a2-73a5-4b8e-9446-fe080a5bb3d7" path="/var/lib/kubelet/pods/059ef4a2-73a5-4b8e-9446-fe080a5bb3d7/volumes" Nov 03 20:56:08.850582 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.850555 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:56:08.932192 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.932120 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f96068d-fb57-4a30-876a-e3158d210c3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f96068d-fb57-4a30-876a-e3158d210c3c-istiod-ca-cert\") pod \"1f96068d-fb57-4a30-876a-e3158d210c3c\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " Nov 03 20:56:08.932192 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.932159 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-podinfo\") pod \"1f96068d-fb57-4a30-876a-e3158d210c3c\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " Nov 03 20:56:08.932395 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.932205 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f96068d-fb57-4a30-876a-e3158d210c3c-kube-api-access-w78l2\" (UniqueName: \"kubernetes.io/projected/1f96068d-fb57-4a30-876a-e3158d210c3c-kube-api-access-w78l2\") pod \"1f96068d-fb57-4a30-876a-e3158d210c3c\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " Nov 03 20:56:08.932395 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.932285 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-envoy\") pod \"1f96068d-fb57-4a30-876a-e3158d210c3c\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " Nov 03 20:56:08.932395 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.932306 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-token\" (UniqueName: \"kubernetes.io/projected/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-token\") pod \"1f96068d-fb57-4a30-876a-e3158d210c3c\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " Nov 03 20:56:08.932492 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.932467 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-data\") pod \"1f96068d-fb57-4a30-876a-e3158d210c3c\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " Nov 03 20:56:08.932551 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.932519 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f96068d-fb57-4a30-876a-e3158d210c3c-istiod-ca-cert" pod "1f96068d-fb57-4a30-876a-e3158d210c3c" (UID: "1f96068d-fb57-4a30-876a-e3158d210c3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:56:08.932551 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.932539 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-workload-certs\") pod \"1f96068d-fb57-4a30-876a-e3158d210c3c\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " Nov 03 20:56:08.932746 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.932717 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-credential-socket\") pod \"1f96068d-fb57-4a30-876a-e3158d210c3c\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " Nov 03 20:56:08.932826 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.932780 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-workload-certs" pod "1f96068d-fb57-4a30-876a-e3158d210c3c" (UID: "1f96068d-fb57-4a30-876a-e3158d210c3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:56:08.932826 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.932790 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-data" pod "1f96068d-fb57-4a30-876a-e3158d210c3c" (UID: "1f96068d-fb57-4a30-876a-e3158d210c3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:56:08.932826 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.932819 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-workload-socket\") pod \"1f96068d-fb57-4a30-876a-e3158d210c3c\" (UID: \"1f96068d-fb57-4a30-876a-e3158d210c3c\") " Nov 03 20:56:08.933000 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.932977 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-credential-socket" pod "1f96068d-fb57-4a30-876a-e3158d210c3c" (UID: "1f96068d-fb57-4a30-876a-e3158d210c3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:56:08.933083 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.933060 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-workload-socket" pod "1f96068d-fb57-4a30-876a-e3158d210c3c" (UID: "1f96068d-fb57-4a30-876a-e3158d210c3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:56:08.933840 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.933812 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:08.933955 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.933843 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:08.933955 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.933868 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:08.933955 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.933882 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:08.933955 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.933895 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f96068d-fb57-4a30-876a-e3158d210c3c-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:08.934566 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.934539 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-podinfo" pod "1f96068d-fb57-4a30-876a-e3158d210c3c" (UID: "1f96068d-fb57-4a30-876a-e3158d210c3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:56:08.934880 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.934856 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-envoy" pod "1f96068d-fb57-4a30-876a-e3158d210c3c" (UID: "1f96068d-fb57-4a30-876a-e3158d210c3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:56:08.934880 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.934873 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f96068d-fb57-4a30-876a-e3158d210c3c-kube-api-access-w78l2" pod "1f96068d-fb57-4a30-876a-e3158d210c3c" (UID: "1f96068d-fb57-4a30-876a-e3158d210c3c"). InnerVolumeSpecName "kube-api-access-w78l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:56:08.935154 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:08.935132 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-token" pod "1f96068d-fb57-4a30-876a-e3158d210c3c" (UID: "1f96068d-fb57-4a30-876a-e3158d210c3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:56:09.035761 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.035724 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:09.035761 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.035753 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w78l2\" (UniqueName: \"kubernetes.io/projected/1f96068d-fb57-4a30-876a-e3158d210c3c-kube-api-access-w78l2\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:09.035761 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.035764 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:09.035982 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.035773 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f96068d-fb57-4a30-876a-e3158d210c3c-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:09.165438 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.165406 4493 generic.go:356] "Generic (PLEG): container finished" podID="1f96068d-fb57-4a30-876a-e3158d210c3c" containerID="b393a9b8a28d83ec6735dc5a54e18f7073e4a9f406519cf338a769ca670c8f7c" exitCode=0 Nov 03 20:56:09.165616 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.165458 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" Nov 03 20:56:09.165616 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.165472 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" event={"ID":"1f96068d-fb57-4a30-876a-e3158d210c3c","Type":"ContainerDied","Data":"b393a9b8a28d83ec6735dc5a54e18f7073e4a9f406519cf338a769ca670c8f7c"} Nov 03 20:56:09.165616 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.165514 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7" event={"ID":"1f96068d-fb57-4a30-876a-e3158d210c3c","Type":"ContainerDied","Data":"9fdb3c66f74e1f3ccc224270f75576abca18d9772a59b55d19b7abf15a049822"} Nov 03 20:56:09.165616 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.165535 4493 scope.go:122] "RemoveContainer" containerID="b393a9b8a28d83ec6735dc5a54e18f7073e4a9f406519cf338a769ca670c8f7c" Nov 03 20:56:09.178773 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.178754 4493 scope.go:122] "RemoveContainer" containerID="eb5d8bdef9431fa1341ba5ed49b46cbf67a0875fde27af1822fd63ba0b5c3702" Nov 03 20:56:09.188726 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.188707 4493 scope.go:122] "RemoveContainer" containerID="b393a9b8a28d83ec6735dc5a54e18f7073e4a9f406519cf338a769ca670c8f7c" Nov 03 20:56:09.189009 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:56:09.188991 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b393a9b8a28d83ec6735dc5a54e18f7073e4a9f406519cf338a769ca670c8f7c\": container with ID starting with b393a9b8a28d83ec6735dc5a54e18f7073e4a9f406519cf338a769ca670c8f7c not found: ID does not exist" containerID="b393a9b8a28d83ec6735dc5a54e18f7073e4a9f406519cf338a769ca670c8f7c" Nov 03 20:56:09.189058 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.189018 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b393a9b8a28d83ec6735dc5a54e18f7073e4a9f406519cf338a769ca670c8f7c"} err="failed to get container status \"b393a9b8a28d83ec6735dc5a54e18f7073e4a9f406519cf338a769ca670c8f7c\": rpc error: code = NotFound desc = could not find container \"b393a9b8a28d83ec6735dc5a54e18f7073e4a9f406519cf338a769ca670c8f7c\": container with ID starting with b393a9b8a28d83ec6735dc5a54e18f7073e4a9f406519cf338a769ca670c8f7c not found: ID does not exist" Nov 03 20:56:09.189058 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.189037 4493 scope.go:122] "RemoveContainer" containerID="eb5d8bdef9431fa1341ba5ed49b46cbf67a0875fde27af1822fd63ba0b5c3702" Nov 03 20:56:09.189267 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:56:09.189251 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb5d8bdef9431fa1341ba5ed49b46cbf67a0875fde27af1822fd63ba0b5c3702\": container with ID starting with eb5d8bdef9431fa1341ba5ed49b46cbf67a0875fde27af1822fd63ba0b5c3702 not found: ID does not exist" containerID="eb5d8bdef9431fa1341ba5ed49b46cbf67a0875fde27af1822fd63ba0b5c3702" Nov 03 20:56:09.189320 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.189274 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb5d8bdef9431fa1341ba5ed49b46cbf67a0875fde27af1822fd63ba0b5c3702"} err="failed to get container status \"eb5d8bdef9431fa1341ba5ed49b46cbf67a0875fde27af1822fd63ba0b5c3702\": rpc error: code = NotFound desc = could not find container \"eb5d8bdef9431fa1341ba5ed49b46cbf67a0875fde27af1822fd63ba0b5c3702\": container with ID starting with eb5d8bdef9431fa1341ba5ed49b46cbf67a0875fde27af1822fd63ba0b5c3702 not found: ID does not exist" Nov 03 20:56:09.326892 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.326855 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7"] Nov 03 20:56:09.330877 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.330844 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-1a55d7db-openshift-default-855876c69-cjgj7"] Nov 03 20:56:09.623888 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.623853 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2"] Nov 03 20:56:09.624512 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.624497 4493 cpu_manager.go:414] "containerMap: removing container" podUID="1f96068d-fb57-4a30-876a-e3158d210c3c" containerName="istio-proxy" Nov 03 20:56:09.624578 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.624513 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f96068d-fb57-4a30-876a-e3158d210c3c" containerName="istio-proxy" Nov 03 20:56:09.624578 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.624543 4493 cpu_manager.go:414] "containerMap: removing container" podUID="1f96068d-fb57-4a30-876a-e3158d210c3c" containerName="istio-proxy" Nov 03 20:56:09.624578 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.624548 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f96068d-fb57-4a30-876a-e3158d210c3c" containerName="istio-proxy" Nov 03 20:56:09.624578 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.624562 4493 cpu_manager.go:414] "containerMap: removing container" podUID="059ef4a2-73a5-4b8e-9446-fe080a5bb3d7" containerName="maas-api" Nov 03 20:56:09.624578 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.624567 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="059ef4a2-73a5-4b8e-9446-fe080a5bb3d7" containerName="maas-api" Nov 03 20:56:09.624729 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.624667 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f96068d-fb57-4a30-876a-e3158d210c3c" containerName="istio-proxy" Nov 03 20:56:09.624729 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.624681 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="059ef4a2-73a5-4b8e-9446-fe080a5bb3d7" containerName="maas-api" Nov 03 20:56:09.629622 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.629597 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.642480 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.642003 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n"] Nov 03 20:56:09.642924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.642905 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f96068d-fb57-4a30-876a-e3158d210c3c" containerName="istio-proxy" Nov 03 20:56:09.646264 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.646247 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.685435 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.685406 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2"] Nov 03 20:56:09.723008 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.722965 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n"] Nov 03 20:56:09.746771 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.746733 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lhn4\" (UniqueName: \"kubernetes.io/projected/197b41b5-4233-40c8-855a-38b91aedf4eb-kube-api-access-7lhn4\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.746923 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.746801 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197b41b5-4233-40c8-855a-38b91aedf4eb-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.746923 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.746866 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.746923 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.746891 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197b41b5-4233-40c8-855a-38b91aedf4eb-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.746923 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.746911 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.747059 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.746926 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.747166 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.747140 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197b41b5-4233-40c8-855a-38b91aedf4eb-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.747211 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.747196 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/197b41b5-4233-40c8-855a-38b91aedf4eb-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.747387 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.747304 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmfnh\" (UniqueName: \"kubernetes.io/projected/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-kube-api-access-nmfnh\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.747387 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.747363 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.747510 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.747490 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197b41b5-4233-40c8-855a-38b91aedf4eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.747571 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.747540 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197b41b5-4233-40c8-855a-38b91aedf4eb-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.747665 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.747635 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.747768 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.747751 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197b41b5-4233-40c8-855a-38b91aedf4eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.747809 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.747790 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.747864 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.747813 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.747918 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.747853 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.747979 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.747922 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197b41b5-4233-40c8-855a-38b91aedf4eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.774768 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.774746 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-spnvt\"" Nov 03 20:56:09.849697 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.849669 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197b41b5-4233-40c8-855a-38b91aedf4eb-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.849844 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.849712 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/197b41b5-4233-40c8-855a-38b91aedf4eb-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.849844 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.849752 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmfnh\" (UniqueName: \"kubernetes.io/projected/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-kube-api-access-nmfnh\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.849844 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.849782 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.850040 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.849876 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197b41b5-4233-40c8-855a-38b91aedf4eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.850040 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.849932 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197b41b5-4233-40c8-855a-38b91aedf4eb-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.850040 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.849981 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.850173 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.850092 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197b41b5-4233-40c8-855a-38b91aedf4eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.850173 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.850101 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197b41b5-4233-40c8-855a-38b91aedf4eb-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.850173 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.850163 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.850322 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.850201 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.850322 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.850228 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.850322 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.850266 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197b41b5-4233-40c8-855a-38b91aedf4eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.850322 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.850305 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197b41b5-4233-40c8-855a-38b91aedf4eb-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.850576 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.850501 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.850576 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.850514 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lhn4\" (UniqueName: \"kubernetes.io/projected/197b41b5-4233-40c8-855a-38b91aedf4eb-kube-api-access-7lhn4\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.850690 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.850609 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197b41b5-4233-40c8-855a-38b91aedf4eb-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.850690 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.850660 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.850847 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.850690 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197b41b5-4233-40c8-855a-38b91aedf4eb-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.850847 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.850725 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.850847 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.850753 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.851009 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.850890 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197b41b5-4233-40c8-855a-38b91aedf4eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.851169 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.851149 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197b41b5-4233-40c8-855a-38b91aedf4eb-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.851237 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.851155 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.851237 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.851201 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.851436 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.851411 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197b41b5-4233-40c8-855a-38b91aedf4eb-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.851510 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.851476 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.851935 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.851911 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.853619 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.853597 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197b41b5-4233-40c8-855a-38b91aedf4eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.853734 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.853660 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.853734 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.853676 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197b41b5-4233-40c8-855a-38b91aedf4eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.853951 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.853926 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.904397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.904288 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/197b41b5-4233-40c8-855a-38b91aedf4eb-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.904397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.904382 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.904808 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.904790 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lhn4\" (UniqueName: \"kubernetes.io/projected/197b41b5-4233-40c8-855a-38b91aedf4eb-kube-api-access-7lhn4\") pod \"maas-default-gateway-openshift-default-696f47c6c-xd49n\" (UID: \"197b41b5-4233-40c8-855a-38b91aedf4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:09.904905 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.904830 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmfnh\" (UniqueName: \"kubernetes.io/projected/da127cda-8ef4-43c8-8eb8-2fe89b52ffcd-kube-api-access-nmfnh\") pod \"maas-default-gateway-openshift-default-696f47c6c-zgpq2\" (UID: \"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.945614 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.945585 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:09.959150 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:09.959122 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:10.394035 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:10.393984 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2"] Nov 03 20:56:10.397923 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:10.397898 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n"] Nov 03 20:56:10.398934 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:10.398895 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:56:10.399031 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:10.398942 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:56:10.399031 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:10.398980 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:56:10.399031 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:10.399019 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:56:10.399031 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:10.399022 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:56:10.399195 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:10.399089 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:56:10.566090 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:10.566057 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f96068d-fb57-4a30-876a-e3158d210c3c" path="/var/lib/kubelet/pods/1f96068d-fb57-4a30-876a-e3158d210c3c/volumes" Nov 03 20:56:11.156742 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:11.156712 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 20:56:11.210360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:11.209571 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" event={"ID":"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd","Type":"ContainerStarted","Data":"0c5b831dae8a8297afa5f5b83fede63d92c035960fdfed5b11c0a054270908c6"} Nov 03 20:56:11.210360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:11.209640 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" event={"ID":"da127cda-8ef4-43c8-8eb8-2fe89b52ffcd","Type":"ContainerStarted","Data":"ad2f29453f6f021b2c9f339ca682cc012c2a6f9f5db80e7f22817028ef489048"} Nov 03 20:56:11.235354 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:11.230383 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" event={"ID":"197b41b5-4233-40c8-855a-38b91aedf4eb","Type":"ContainerStarted","Data":"e474a313566ce0b24f5f2b9c1b63eb292e028faf6fad209897a8a4772428dba6"} Nov 03 20:56:11.235354 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:11.230455 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" event={"ID":"197b41b5-4233-40c8-855a-38b91aedf4eb","Type":"ContainerStarted","Data":"d770b02b4bf08ff0ebf5c46349b50624df2a92fb1051b5ddfa6c1e16a4c84a34"} Nov 03 20:56:11.537154 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:11.537035 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" podStartSLOduration=2.537014986 podStartE2EDuration="2.537014986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:11.424021844 +0000 UTC m=+1642.982012366" watchObservedRunningTime="2026-11-03 20:56:11.537014986 +0000 UTC m=+1643.095005487" Nov 03 20:56:11.539931 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:11.539878 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" podStartSLOduration=2.539861238 podStartE2EDuration="2.539861238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:11.534804817 +0000 UTC m=+1643.092795319" watchObservedRunningTime="2026-11-03 20:56:11.539861238 +0000 UTC m=+1643.097851738" Nov 03 20:56:11.946383 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:11.946140 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:11.947273 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:11.947255 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-zgpq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:56:11.947326 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:11.947292 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" podUID="da127cda-8ef4-43c8-8eb8-2fe89b52ffcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 20:56:11.960315 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:11.960291 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:11.961447 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:11.961424 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-xd49n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:56:11.961558 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:11.961472 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" podUID="197b41b5-4233-40c8-855a-38b91aedf4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 20:56:12.946540 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:12.946511 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-zgpq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:56:12.946975 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:12.946949 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" podUID="da127cda-8ef4-43c8-8eb8-2fe89b52ffcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 20:56:12.959802 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:12.959781 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-xd49n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:56:12.959992 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:12.959974 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" podUID="197b41b5-4233-40c8-855a-38b91aedf4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 20:56:13.946606 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:13.946573 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-zgpq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:56:13.947000 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:13.946625 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" podUID="da127cda-8ef4-43c8-8eb8-2fe89b52ffcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 20:56:13.960476 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:13.960450 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-xd49n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:56:13.960599 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:13.960496 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" podUID="197b41b5-4233-40c8-855a-38b91aedf4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 20:56:14.946704 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:14.946674 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-zgpq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:56:14.947200 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:14.947174 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" podUID="da127cda-8ef4-43c8-8eb8-2fe89b52ffcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 20:56:14.960030 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:14.959988 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-xd49n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:56:14.960119 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:14.960041 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" podUID="197b41b5-4233-40c8-855a-38b91aedf4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 20:56:15.946602 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:15.946561 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-zgpq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:56:15.946785 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:15.946624 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" podUID="da127cda-8ef4-43c8-8eb8-2fe89b52ffcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 20:56:15.959495 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:15.959459 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-xd49n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:56:15.959636 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:15.959506 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" podUID="197b41b5-4233-40c8-855a-38b91aedf4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 20:56:16.189395 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:16.187483 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt"] Nov 03 20:56:16.189395 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:16.187749 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" podUID="9f4c678e-8285-4d16-984d-39fa208f1ae8" containerName="istio-proxy" containerID="cri-o://25a222b086d47718c2e12b83bcfb777181cc5c315736b175d023640e33330fbe" gracePeriod=30 Nov 03 20:56:16.946143 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:16.946106 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-zgpq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:56:16.946379 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:16.946169 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" podUID="da127cda-8ef4-43c8-8eb8-2fe89b52ffcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 20:56:16.959684 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:16.959649 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-xd49n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:56:16.960007 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:16.959717 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" podUID="197b41b5-4233-40c8-855a-38b91aedf4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 20:56:17.946419 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:17.946379 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-zgpq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:56:17.946632 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:17.946448 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" podUID="da127cda-8ef4-43c8-8eb8-2fe89b52ffcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 20:56:17.959740 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:17.959713 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-xd49n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:56:17.960053 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:17.959765 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" podUID="197b41b5-4233-40c8-855a-38b91aedf4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 20:56:19.187354 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:19.187308 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:19.189824 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:19.189788 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:19.190919 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:19.190898 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" Nov 03 20:56:19.235176 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:19.235142 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:19.235600 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:19.235581 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:19.236480 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:19.236427 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" Nov 03 20:56:19.337981 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:19.337951 4493 patch_prober.go:29] interesting pod/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:56:19.338135 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:19.338013 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" podUID="9f4c678e-8285-4d16-984d-39fa208f1ae8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:56:21.424761 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.424732 4493 generic.go:356] "Generic (PLEG): container finished" podID="9f4c678e-8285-4d16-984d-39fa208f1ae8" containerID="25a222b086d47718c2e12b83bcfb777181cc5c315736b175d023640e33330fbe" exitCode=0 Nov 03 20:56:21.425251 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.424817 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" event={"ID":"9f4c678e-8285-4d16-984d-39fa208f1ae8","Type":"ContainerDied","Data":"25a222b086d47718c2e12b83bcfb777181cc5c315736b175d023640e33330fbe"} Nov 03 20:56:21.425251 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.424902 4493 scope.go:122] "RemoveContainer" containerID="1a4e7de59a82a80ef3ef355cd595031cf005610156f6488827b292e41cc035ee" Nov 03 20:56:21.476043 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.475985 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:56:21.499362 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.499304 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-workload-certs\") pod \"9f4c678e-8285-4d16-984d-39fa208f1ae8\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " Nov 03 20:56:21.499507 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.499396 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-workload-socket\") pod \"9f4c678e-8285-4d16-984d-39fa208f1ae8\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " Nov 03 20:56:21.499507 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.499494 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f4c678e-8285-4d16-984d-39fa208f1ae8-kube-api-access-lpvjm\" (UniqueName: \"kubernetes.io/projected/9f4c678e-8285-4d16-984d-39fa208f1ae8-kube-api-access-lpvjm\") pod \"9f4c678e-8285-4d16-984d-39fa208f1ae8\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " Nov 03 20:56:21.499629 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.499578 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-data\") pod \"9f4c678e-8285-4d16-984d-39fa208f1ae8\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " Nov 03 20:56:21.499629 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.499581 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-workload-certs" pod "9f4c678e-8285-4d16-984d-39fa208f1ae8" (UID: "9f4c678e-8285-4d16-984d-39fa208f1ae8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:56:21.499629 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.499626 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-token\" (UniqueName: \"kubernetes.io/projected/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-token\") pod \"9f4c678e-8285-4d16-984d-39fa208f1ae8\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " Nov 03 20:56:21.499802 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.499669 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-credential-socket\") pod \"9f4c678e-8285-4d16-984d-39fa208f1ae8\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " Nov 03 20:56:21.499802 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.499732 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-workload-socket" pod "9f4c678e-8285-4d16-984d-39fa208f1ae8" (UID: "9f4c678e-8285-4d16-984d-39fa208f1ae8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:56:21.499961 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.499840 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-podinfo\") pod \"9f4c678e-8285-4d16-984d-39fa208f1ae8\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " Nov 03 20:56:21.499961 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.499908 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-data" pod "9f4c678e-8285-4d16-984d-39fa208f1ae8" (UID: "9f4c678e-8285-4d16-984d-39fa208f1ae8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:56:21.499961 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.499940 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-envoy\") pod \"9f4c678e-8285-4d16-984d-39fa208f1ae8\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " Nov 03 20:56:21.500102 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.499963 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-credential-socket" pod "9f4c678e-8285-4d16-984d-39fa208f1ae8" (UID: "9f4c678e-8285-4d16-984d-39fa208f1ae8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:56:21.500102 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.499996 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f4c678e-8285-4d16-984d-39fa208f1ae8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4c678e-8285-4d16-984d-39fa208f1ae8-istiod-ca-cert\") pod \"9f4c678e-8285-4d16-984d-39fa208f1ae8\" (UID: \"9f4c678e-8285-4d16-984d-39fa208f1ae8\") " Nov 03 20:56:21.500680 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.500652 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f4c678e-8285-4d16-984d-39fa208f1ae8-istiod-ca-cert" pod "9f4c678e-8285-4d16-984d-39fa208f1ae8" (UID: "9f4c678e-8285-4d16-984d-39fa208f1ae8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:56:21.501124 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.501098 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:21.501224 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.501130 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:21.501224 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.501146 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4c678e-8285-4d16-984d-39fa208f1ae8-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:21.501224 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.501161 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:21.501224 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.501174 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:21.502051 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.502025 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4c678e-8285-4d16-984d-39fa208f1ae8-kube-api-access-lpvjm" pod "9f4c678e-8285-4d16-984d-39fa208f1ae8" (UID: "9f4c678e-8285-4d16-984d-39fa208f1ae8"). InnerVolumeSpecName "kube-api-access-lpvjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:56:21.502051 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.502041 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-token" pod "9f4c678e-8285-4d16-984d-39fa208f1ae8" (UID: "9f4c678e-8285-4d16-984d-39fa208f1ae8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:56:21.502173 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.502100 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-podinfo" pod "9f4c678e-8285-4d16-984d-39fa208f1ae8" (UID: "9f4c678e-8285-4d16-984d-39fa208f1ae8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:56:21.502297 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.502281 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-envoy" pod "9f4c678e-8285-4d16-984d-39fa208f1ae8" (UID: "9f4c678e-8285-4d16-984d-39fa208f1ae8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:56:21.602814 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.602780 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:21.602814 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.602813 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:21.603003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.602830 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpvjm\" (UniqueName: \"kubernetes.io/projected/9f4c678e-8285-4d16-984d-39fa208f1ae8-kube-api-access-lpvjm\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:21.603003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:21.602846 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4c678e-8285-4d16-984d-39fa208f1ae8-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:21.746240 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:56:21.746156 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:56:22.433497 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:22.433467 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" Nov 03 20:56:22.433938 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:22.433492 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt" event={"ID":"9f4c678e-8285-4d16-984d-39fa208f1ae8","Type":"ContainerDied","Data":"1809a30a603767ce6192d5b96d5ab0eeaac145c9615a899a1db6c04277fe092e"} Nov 03 20:56:22.433938 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:22.433592 4493 scope.go:122] "RemoveContainer" containerID="25a222b086d47718c2e12b83bcfb777181cc5c315736b175d023640e33330fbe" Nov 03 20:56:22.637569 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:22.637532 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt"] Nov 03 20:56:22.683858 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:22.683785 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e3e5cd62-openshift-default-56978d95df-2l8wt"] Nov 03 20:56:24.565268 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.565236 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f4c678e-8285-4d16-984d-39fa208f1ae8" path="/var/lib/kubelet/pods/9f4c678e-8285-4d16-984d-39fa208f1ae8/volumes" Nov 03 20:56:24.589405 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.589374 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k"] Nov 03 20:56:24.590085 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.590068 4493 cpu_manager.go:414] "containerMap: removing container" podUID="9f4c678e-8285-4d16-984d-39fa208f1ae8" containerName="istio-proxy" Nov 03 20:56:24.590085 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.590085 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f4c678e-8285-4d16-984d-39fa208f1ae8" containerName="istio-proxy" Nov 03 20:56:24.590244 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.590233 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f4c678e-8285-4d16-984d-39fa208f1ae8" containerName="istio-proxy" Nov 03 20:56:24.595038 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.595018 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.653836 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.653804 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.654001 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.653902 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.654043 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.654016 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.654106 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.654086 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7979\" (UniqueName: \"kubernetes.io/projected/7bae91ea-007d-4af0-861d-9e7fc6610997-kube-api-access-n7979\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.654213 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.654195 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bae91ea-007d-4af0-861d-9e7fc6610997-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.654409 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.654389 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.654646 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.654622 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.654776 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.654721 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.654776 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.654755 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.698823 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.698795 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k"] Nov 03 20:56:24.756368 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.756321 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.756536 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.756376 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.756536 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.756448 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.756536 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.756504 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.756718 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.756538 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.756718 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.756616 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7979\" (UniqueName: \"kubernetes.io/projected/7bae91ea-007d-4af0-861d-9e7fc6610997-kube-api-access-n7979\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.756819 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.756722 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bae91ea-007d-4af0-861d-9e7fc6610997-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.756950 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.756866 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.757033 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.756879 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.757033 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.756912 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.757117 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.757090 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.757117 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.757091 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.757528 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.757507 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bae91ea-007d-4af0-861d-9e7fc6610997-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.757575 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.757538 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.759235 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.759216 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.759932 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.759886 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.809917 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.809885 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.810154 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.810135 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7979\" (UniqueName: \"kubernetes.io/projected/7bae91ea-007d-4af0-861d-9e7fc6610997-kube-api-access-n7979\") pod \"maas-default-gateway-openshift-default-696f47c6c-s4r2k\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:24.908466 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:24.908392 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:25.341060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:25.341024 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k"] Nov 03 20:56:25.385063 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:56:25.385034 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bae91ea_007d_4af0_861d_9e7fc6610997.slice/crio-328d1a247a508232a1119f6f17071c6750886db5dae5c0fe95f85ef46fa1728a WatchSource:0}: Error finding container 328d1a247a508232a1119f6f17071c6750886db5dae5c0fe95f85ef46fa1728a: Status 404 returned error can't find the container with id 328d1a247a508232a1119f6f17071c6750886db5dae5c0fe95f85ef46fa1728a Nov 03 20:56:25.387817 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:25.387785 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:56:25.387912 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:25.387854 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:56:25.387912 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:25.387881 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:56:25.473619 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:25.473585 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" event={"ID":"7bae91ea-007d-4af0-861d-9e7fc6610997","Type":"ContainerStarted","Data":"f075e686dcae4c2b4dc3809237934e3930079f7b9a97b053dc038bfd09915d5f"} Nov 03 20:56:25.473755 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:25.473677 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" event={"ID":"7bae91ea-007d-4af0-861d-9e7fc6610997","Type":"ContainerStarted","Data":"328d1a247a508232a1119f6f17071c6750886db5dae5c0fe95f85ef46fa1728a"} Nov 03 20:56:25.594948 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:25.594830 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" podStartSLOduration=1.594806945 podStartE2EDuration="1.594806945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:25.588837212 +0000 UTC m=+1657.146827735" watchObservedRunningTime="2026-11-03 20:56:25.594806945 +0000 UTC m=+1657.152797445" Nov 03 20:56:25.909000 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:25.908919 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:26.911454 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:26.911413 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-s4r2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:56:26.911898 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:26.911474 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" podUID="7bae91ea-007d-4af0-861d-9e7fc6610997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 20:56:27.909744 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:27.909710 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-s4r2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:56:27.909927 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:27.909762 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" podUID="7bae91ea-007d-4af0-861d-9e7fc6610997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 20:56:28.909151 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:28.909066 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-s4r2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:56:28.909151 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:28.909126 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" podUID="7bae91ea-007d-4af0-861d-9e7fc6610997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 20:56:29.911871 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:29.911834 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-s4r2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:56:29.912449 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:29.911898 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" podUID="7bae91ea-007d-4af0-861d-9e7fc6610997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 20:56:30.908861 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:30.908824 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-s4r2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:56:30.909042 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:30.908879 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" podUID="7bae91ea-007d-4af0-861d-9e7fc6610997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 20:56:31.908733 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:31.908696 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-s4r2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:56:31.909160 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:31.908762 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" podUID="7bae91ea-007d-4af0-861d-9e7fc6610997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 20:56:32.909253 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:32.909222 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-s4r2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:56:32.909670 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:32.909274 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" podUID="7bae91ea-007d-4af0-861d-9e7fc6610997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 20:56:34.520141 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:34.520099 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:34.521651 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:34.521616 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:34.522122 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:56:34.522095 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 20:56:51.747769 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:56:51.747726 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:57:20.913491 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:20.913452 4493 patch_prober.go:29] interesting pod/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:57:21.002879 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:20.913521 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" podUID="0d3000f1-172d-4b7b-9d45-fa04ef63145f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:57:21.750720 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:57:21.750682 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:57:22.229385 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:22.229349 4493 generic.go:356] "Generic (PLEG): container finished" podID="0d3000f1-172d-4b7b-9d45-fa04ef63145f" containerID="ec76955c5127e08a3ce1b7f2d997987b8d7360848ce06ce1a81ce2fcb71cc88a" exitCode=0 Nov 03 20:57:22.229385 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:22.229361 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" event={"ID":"0d3000f1-172d-4b7b-9d45-fa04ef63145f","Type":"ContainerDied","Data":"ec76955c5127e08a3ce1b7f2d997987b8d7360848ce06ce1a81ce2fcb71cc88a"} Nov 03 20:57:22.229902 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:22.229884 4493 scope.go:122] "RemoveContainer" containerID="ec76955c5127e08a3ce1b7f2d997987b8d7360848ce06ce1a81ce2fcb71cc88a" Nov 03 20:57:22.231485 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:22.231451 4493 generic.go:356] "Generic (PLEG): container finished" podID="a47bd78c-2c95-4eb1-a2a3-b151a424ebdc" containerID="5d044bb04f3c6228a831e8fbf88f5adfbab9956ac9bac3af9fbf979d3ce28463" exitCode=0 Nov 03 20:57:22.231617 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:22.231500 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" event={"ID":"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc","Type":"ContainerDied","Data":"5d044bb04f3c6228a831e8fbf88f5adfbab9956ac9bac3af9fbf979d3ce28463"} Nov 03 20:57:22.231884 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:22.231859 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:57:22.231940 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:22.231926 4493 scope.go:122] "RemoveContainer" containerID="5d044bb04f3c6228a831e8fbf88f5adfbab9956ac9bac3af9fbf979d3ce28463" Nov 03 20:57:22.231985 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:22.231940 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:57:22.232020 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:22.231986 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:57:22.233503 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:22.233473 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:57:22.233610 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:22.233539 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:57:22.233610 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:22.233574 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:57:23.265414 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:23.265367 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" event={"ID":"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc","Type":"ContainerStarted","Data":"4aebf80031f920901e827482741b42c35b7f3cfd81268bda150b6039541523f4"} Nov 03 20:57:23.267845 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:23.267804 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" event={"ID":"0d3000f1-172d-4b7b-9d45-fa04ef63145f","Type":"ContainerStarted","Data":"470fd3c29321dd66952480f1ca289ec06aa43e0b0d3bc6ea90f11048caadd103"} Nov 03 20:57:23.705360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:23.703763 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:57:23.705360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:23.704616 4493 patch_prober.go:29] interesting pod/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 20:57:23.705360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:23.704702 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" podUID="0d3000f1-172d-4b7b-9d45-fa04ef63145f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 20:57:23.748048 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:23.747846 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:57:23.750083 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:23.750057 4493 patch_prober.go:29] interesting pod/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:57:23.750179 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:23.750108 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" podUID="a47bd78c-2c95-4eb1-a2a3-b151a424ebdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 20:57:24.703360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:24.695722 4493 patch_prober.go:29] interesting pod/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 20:57:24.703360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:24.695799 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" podUID="0d3000f1-172d-4b7b-9d45-fa04ef63145f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 20:57:24.748038 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:24.748001 4493 patch_prober.go:29] interesting pod/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:57:24.748206 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:24.748056 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" podUID="a47bd78c-2c95-4eb1-a2a3-b151a424ebdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 20:57:24.751839 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:24.751816 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:57:25.704359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:25.703807 4493 patch_prober.go:29] interesting pod/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 20:57:25.704359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:25.703879 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" podUID="0d3000f1-172d-4b7b-9d45-fa04ef63145f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 20:57:25.747841 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:25.747802 4493 patch_prober.go:29] interesting pod/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:57:25.748013 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:25.747872 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" podUID="a47bd78c-2c95-4eb1-a2a3-b151a424ebdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 20:57:26.705362 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:26.696744 4493 patch_prober.go:29] interesting pod/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 20:57:26.705362 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:26.696806 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" podUID="0d3000f1-172d-4b7b-9d45-fa04ef63145f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 20:57:26.747657 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:26.747312 4493 patch_prober.go:29] interesting pod/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:57:26.747657 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:26.747402 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" podUID="a47bd78c-2c95-4eb1-a2a3-b151a424ebdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 20:57:27.695360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:27.693670 4493 patch_prober.go:29] interesting pod/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 20:57:27.695360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:27.693730 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" podUID="0d3000f1-172d-4b7b-9d45-fa04ef63145f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 20:57:27.747199 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:27.747166 4493 patch_prober.go:29] interesting pod/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:57:27.747617 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:27.747223 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" podUID="a47bd78c-2c95-4eb1-a2a3-b151a424ebdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 20:57:28.592152 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:28.592114 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt"] Nov 03 20:57:28.593192 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:28.593165 4493 cpu_manager.go:414] "containerMap: removing container" podUID="9f4c678e-8285-4d16-984d-39fa208f1ae8" containerName="istio-proxy" Nov 03 20:57:28.593192 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:28.593192 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f4c678e-8285-4d16-984d-39fa208f1ae8" containerName="istio-proxy" Nov 03 20:57:28.593409 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:28.593392 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f4c678e-8285-4d16-984d-39fa208f1ae8" containerName="istio-proxy" Nov 03 20:57:28.597440 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:28.597417 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt" Nov 03 20:57:28.648047 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:28.648013 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt"] Nov 03 20:57:28.689664 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:28.689635 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-45f792-serving-cert\"" Nov 03 20:57:28.698050 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:28.698022 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:57:28.698467 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:28.698447 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:57:28.699179 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:28.699155 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:57:28.758013 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:28.757985 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:57:28.759239 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:28.759219 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:57:28.785396 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:28.785369 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rs62\" (UniqueName: \"kubernetes.io/projected/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-kube-api-access-7rs62\") pod \"maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt\" (UID: \"3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt" Nov 03 20:57:28.785566 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:28.785518 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt\" (UID: \"3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt" Nov 03 20:57:28.785780 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:28.785764 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt\" (UID: \"3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt" Nov 03 20:57:28.785872 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:28.785848 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt\" (UID: \"3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt" Nov 03 20:57:28.889924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:28.889142 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt\" (UID: \"3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt" Nov 03 20:57:28.889924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:28.889310 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rs62\" (UniqueName: \"kubernetes.io/projected/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-kube-api-access-7rs62\") pod \"maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt\" (UID: \"3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt" Nov 03 20:57:28.889924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:28.889708 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt\" (UID: \"3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt" Nov 03 20:57:28.890312 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:28.890111 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt\" (UID: \"3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt" Nov 03 20:57:28.890400 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:57:28.890352 4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-serving-cert: secret "maas-api-e2e-shared-a-w3-45f792-serving-cert" not found Nov 03 20:57:28.890458 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:57:28.890413 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-maas-api-tls podName:3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8 nodeName:}" failed. No retries permitted until 2026-11-03 20:57:29.390393474 +0000 UTC m=+1720.948383955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-maas-api-tls") pod "maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt" (UID: "3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8") : secret "maas-api-e2e-shared-a-w3-45f792-serving-cert" not found Nov 03 20:57:28.890458 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:28.890442 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt\" (UID: \"3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt" Nov 03 20:57:28.897540 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:28.894050 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt\" (UID: \"3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt" Nov 03 20:57:28.943442 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:28.943412 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rs62\" (UniqueName: \"kubernetes.io/projected/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-kube-api-access-7rs62\") pod \"maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt\" (UID: \"3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt" Nov 03 20:57:29.408222 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:29.408184 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt\" (UID: \"3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt" Nov 03 20:57:29.410960 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:29.410935 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt\" (UID: \"3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt" Nov 03 20:57:29.517447 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:29.517414 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt" Nov 03 20:57:29.894161 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:29.894129 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt"] Nov 03 20:57:29.897615 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:57:29.897579 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b85c1d1_26ab_4496_8721_fcf0b4ff3ff8.slice/crio-4401bfe082976c6bfb730d1a6ca7543ed9de31be931b90e19c1cb026a6eb4ee7 WatchSource:0}: Error finding container 4401bfe082976c6bfb730d1a6ca7543ed9de31be931b90e19c1cb026a6eb4ee7: Status 404 returned error can't find the container with id 4401bfe082976c6bfb730d1a6ca7543ed9de31be931b90e19c1cb026a6eb4ee7 Nov 03 20:57:30.364408 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:30.364370 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt" event={"ID":"3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8","Type":"ContainerStarted","Data":"4401bfe082976c6bfb730d1a6ca7543ed9de31be931b90e19c1cb026a6eb4ee7"} Nov 03 20:57:31.373004 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:31.372967 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt" event={"ID":"3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8","Type":"ContainerStarted","Data":"d426af98f29456e34761cadddbb23e28574a3a2087c8bdc276c67b74a3e3cf65"} Nov 03 20:57:31.373445 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:31.373065 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt" Nov 03 20:57:31.484597 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:31.484545 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt" podStartSLOduration=2.542182818 podStartE2EDuration="3.484530229s" totalImagesPullingTime="942.347411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:31.480733864 +0000 UTC m=+1723.038724367" watchObservedRunningTime="2026-11-03 20:57:31.484530229 +0000 UTC m=+1723.042520726" Nov 03 20:57:35.935377 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:35.935325 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz"] Nov 03 20:57:37.408357 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:37.406983 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt" Nov 03 20:57:40.936156 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:40.936097 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz" podUID="695ce88a-f841-4e61-b76a-d1a0f2b10136" containerName="payload-pre-processing" containerID="cri-o://74fabb929a7ebcc51c3be1588e34041d4c56b1ce883adf7107b6fe70bfefb497" gracePeriod=25 Nov 03 20:57:41.213648 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:41.213575 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz" Nov 03 20:57:41.271026 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:41.270992 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/695ce88a-f841-4e61-b76a-d1a0f2b10136-kube-api-access-jfrbl\" (UniqueName: \"kubernetes.io/projected/695ce88a-f841-4e61-b76a-d1a0f2b10136-kube-api-access-jfrbl\") pod \"695ce88a-f841-4e61-b76a-d1a0f2b10136\" (UID: \"695ce88a-f841-4e61-b76a-d1a0f2b10136\") " Nov 03 20:57:41.271211 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:41.271074 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/695ce88a-f841-4e61-b76a-d1a0f2b10136-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695ce88a-f841-4e61-b76a-d1a0f2b10136-plugins-config-volume\") pod \"695ce88a-f841-4e61-b76a-d1a0f2b10136\" (UID: \"695ce88a-f841-4e61-b76a-d1a0f2b10136\") " Nov 03 20:57:41.271493 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:41.271468 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695ce88a-f841-4e61-b76a-d1a0f2b10136-plugins-config-volume" pod "695ce88a-f841-4e61-b76a-d1a0f2b10136" (UID: "695ce88a-f841-4e61-b76a-d1a0f2b10136"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:41.271885 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:41.271865 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695ce88a-f841-4e61-b76a-d1a0f2b10136-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:41.273379 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:41.273358 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695ce88a-f841-4e61-b76a-d1a0f2b10136-kube-api-access-jfrbl" pod "695ce88a-f841-4e61-b76a-d1a0f2b10136" (UID: "695ce88a-f841-4e61-b76a-d1a0f2b10136"). InnerVolumeSpecName "kube-api-access-jfrbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:41.373828 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:41.373789 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfrbl\" (UniqueName: \"kubernetes.io/projected/695ce88a-f841-4e61-b76a-d1a0f2b10136-kube-api-access-jfrbl\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:41.484940 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:41.484851 4493 generic.go:356] "Generic (PLEG): container finished" podID="695ce88a-f841-4e61-b76a-d1a0f2b10136" containerID="74fabb929a7ebcc51c3be1588e34041d4c56b1ce883adf7107b6fe70bfefb497" exitCode=0 Nov 03 20:57:41.484940 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:41.484903 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz" Nov 03 20:57:41.485142 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:41.484939 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz" event={"ID":"695ce88a-f841-4e61-b76a-d1a0f2b10136","Type":"ContainerDied","Data":"74fabb929a7ebcc51c3be1588e34041d4c56b1ce883adf7107b6fe70bfefb497"} Nov 03 20:57:41.485142 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:41.484997 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz" event={"ID":"695ce88a-f841-4e61-b76a-d1a0f2b10136","Type":"ContainerDied","Data":"b991c5594492dcbc4cbc9d642dd6aa9c106c8d1066342925fa17c17e13a89660"} Nov 03 20:57:41.485142 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:41.485021 4493 scope.go:122] "RemoveContainer" containerID="74fabb929a7ebcc51c3be1588e34041d4c56b1ce883adf7107b6fe70bfefb497" Nov 03 20:57:41.496439 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:41.496417 4493 scope.go:122] "RemoveContainer" containerID="74fabb929a7ebcc51c3be1588e34041d4c56b1ce883adf7107b6fe70bfefb497" Nov 03 20:57:41.496708 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:57:41.496688 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74fabb929a7ebcc51c3be1588e34041d4c56b1ce883adf7107b6fe70bfefb497\": container with ID starting with 74fabb929a7ebcc51c3be1588e34041d4c56b1ce883adf7107b6fe70bfefb497 not found: ID does not exist" containerID="74fabb929a7ebcc51c3be1588e34041d4c56b1ce883adf7107b6fe70bfefb497" Nov 03 20:57:41.496778 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:41.496719 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74fabb929a7ebcc51c3be1588e34041d4c56b1ce883adf7107b6fe70bfefb497"} err="failed to get container status \"74fabb929a7ebcc51c3be1588e34041d4c56b1ce883adf7107b6fe70bfefb497\": rpc error: code = NotFound desc = could not find container \"74fabb929a7ebcc51c3be1588e34041d4c56b1ce883adf7107b6fe70bfefb497\": container with ID starting with 74fabb929a7ebcc51c3be1588e34041d4c56b1ce883adf7107b6fe70bfefb497 not found: ID does not exist" Nov 03 20:57:41.645462 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:41.645429 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz"] Nov 03 20:57:41.649515 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:41.649488 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e8374c4-78747c685f-46sgz"] Nov 03 20:57:42.567368 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:42.567314 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="695ce88a-f841-4e61-b76a-d1a0f2b10136" path="/var/lib/kubelet/pods/695ce88a-f841-4e61-b76a-d1a0f2b10136/volumes" Nov 03 20:57:51.747532 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:57:51.747492 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:57:55.771271 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:55.771236 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8"] Nov 03 20:57:55.881436 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:55.881400 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz"] Nov 03 20:57:58.138206 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.138172 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n"] Nov 03 20:57:58.138913 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.138897 4493 cpu_manager.go:414] "containerMap: removing container" podUID="695ce88a-f841-4e61-b76a-d1a0f2b10136" containerName="payload-pre-processing" Nov 03 20:57:58.138975 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.138914 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="695ce88a-f841-4e61-b76a-d1a0f2b10136" containerName="payload-pre-processing" Nov 03 20:57:58.139029 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.139020 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="695ce88a-f841-4e61-b76a-d1a0f2b10136" containerName="payload-pre-processing" Nov 03 20:57:58.144160 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.144144 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.193623 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.193595 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-211d68bf-openshift-default-dockercfg-c9mrc\"" Nov 03 20:57:58.194065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.194036 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.194249 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.194117 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.194249 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.194216 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.194434 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.194411 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701f971e-5fbb-4f4b-81ec-1dfa66005128-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.194522 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.194505 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.194685 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.194667 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.194802 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.194750 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.194802 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.194799 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.195055 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.194985 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g26wv\" (UniqueName: \"kubernetes.io/projected/701f971e-5fbb-4f4b-81ec-1dfa66005128-kube-api-access-g26wv\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.246467 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.246433 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n"] Nov 03 20:57:58.308860 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.308815 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.309162 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.308942 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.309162 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.308997 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.309162 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.309089 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701f971e-5fbb-4f4b-81ec-1dfa66005128-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.309162 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.309129 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.309536 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.309295 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.309536 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.309378 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.309536 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.309409 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.309654 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.309590 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g26wv\" (UniqueName: \"kubernetes.io/projected/701f971e-5fbb-4f4b-81ec-1dfa66005128-kube-api-access-g26wv\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.310488 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.310446 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.310648 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.310506 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.310812 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.310777 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.312251 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.312153 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701f971e-5fbb-4f4b-81ec-1dfa66005128-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.314968 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.314943 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.321838 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.318892 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.329413 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.323193 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.364149 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.364118 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.364348 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.364259 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g26wv\" (UniqueName: \"kubernetes.io/projected/701f971e-5fbb-4f4b-81ec-1dfa66005128-kube-api-access-g26wv\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lcn7n\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.458748 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.458640 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:57:58.912863 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.912828 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n"] Nov 03 20:57:58.920080 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:57:58.920038 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod701f971e_5fbb_4f4b_81ec_1dfa66005128.slice/crio-e503d470afe4db80a3011ed18ebb0df8ceb336493ddf7d71724edde29a1b98c1 WatchSource:0}: Error finding container e503d470afe4db80a3011ed18ebb0df8ceb336493ddf7d71724edde29a1b98c1: Status 404 returned error can't find the container with id e503d470afe4db80a3011ed18ebb0df8ceb336493ddf7d71724edde29a1b98c1 Nov 03 20:57:58.923261 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.923221 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:57:58.923391 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.923311 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:57:58.923391 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:58.923366 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:57:59.657561 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:59.657519 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" event={"ID":"701f971e-5fbb-4f4b-81ec-1dfa66005128","Type":"ContainerStarted","Data":"73764b575f50c5e1e9b4d1ff2f3b97c84f749f0cfe430d47e02c385ea31812a8"} Nov 03 20:57:59.657965 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:59.657572 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" event={"ID":"701f971e-5fbb-4f4b-81ec-1dfa66005128","Type":"ContainerStarted","Data":"e503d470afe4db80a3011ed18ebb0df8ceb336493ddf7d71724edde29a1b98c1"} Nov 03 20:57:59.814520 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:57:59.814459 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" podStartSLOduration=1.81444473 podStartE2EDuration="1.81444473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:59.813736854 +0000 UTC m=+1751.371727352" watchObservedRunningTime="2026-11-03 20:57:59.81444473 +0000 UTC m=+1751.372435229" Nov 03 20:58:00.459106 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:00.459075 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:58:00.464942 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:00.464917 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:58:00.664487 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:00.664461 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:58:00.665615 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:00.665591 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:58:00.772313 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:00.772216 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8" podUID="f0ae7b36-9635-4ea7-badb-a5a97321c841" containerName="payload-processing" containerID="cri-o://2934f102559eb5d5832546fe5b39fdb7dbe80335dfa4281fe9dac4d845da9816" gracePeriod=25 Nov 03 20:58:00.886579 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:00.886440 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz" podUID="d59bd1c6-7591-4470-b328-b0d60fb6f292" containerName="payload-pre-processing" containerID="cri-o://57a6a75431b24b9a0de9ca3ae3cece7137ae8a0a99dd3efe08d9746a9bb9e8d9" gracePeriod=25 Nov 03 20:58:01.100009 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.099971 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8" Nov 03 20:58:01.160157 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.160126 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0ae7b36-9635-4ea7-badb-a5a97321c841-kube-api-access-l5jhw\" (UniqueName: \"kubernetes.io/projected/f0ae7b36-9635-4ea7-badb-a5a97321c841-kube-api-access-l5jhw\") pod \"f0ae7b36-9635-4ea7-badb-a5a97321c841\" (UID: \"f0ae7b36-9635-4ea7-badb-a5a97321c841\") " Nov 03 20:58:01.160481 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.160425 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0ae7b36-9635-4ea7-badb-a5a97321c841-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ae7b36-9635-4ea7-badb-a5a97321c841-plugins-config-volume\") pod \"f0ae7b36-9635-4ea7-badb-a5a97321c841\" (UID: \"f0ae7b36-9635-4ea7-badb-a5a97321c841\") " Nov 03 20:58:01.161010 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.160975 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ae7b36-9635-4ea7-badb-a5a97321c841-plugins-config-volume" pod "f0ae7b36-9635-4ea7-badb-a5a97321c841" (UID: "f0ae7b36-9635-4ea7-badb-a5a97321c841"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:01.161127 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.161098 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ae7b36-9635-4ea7-badb-a5a97321c841-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:01.164302 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.164043 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ae7b36-9635-4ea7-badb-a5a97321c841-kube-api-access-l5jhw" pod "f0ae7b36-9635-4ea7-badb-a5a97321c841" (UID: "f0ae7b36-9635-4ea7-badb-a5a97321c841"). InnerVolumeSpecName "kube-api-access-l5jhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:01.180892 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.180862 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz" Nov 03 20:58:01.261692 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.261660 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d59bd1c6-7591-4470-b328-b0d60fb6f292-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d59bd1c6-7591-4470-b328-b0d60fb6f292-plugins-config-volume\") pod \"d59bd1c6-7591-4470-b328-b0d60fb6f292\" (UID: \"d59bd1c6-7591-4470-b328-b0d60fb6f292\") " Nov 03 20:58:01.261944 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.261928 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d59bd1c6-7591-4470-b328-b0d60fb6f292-kube-api-access-vtvm2\" (UniqueName: \"kubernetes.io/projected/d59bd1c6-7591-4470-b328-b0d60fb6f292-kube-api-access-vtvm2\") pod \"d59bd1c6-7591-4470-b328-b0d60fb6f292\" (UID: \"d59bd1c6-7591-4470-b328-b0d60fb6f292\") " Nov 03 20:58:01.262038 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.262012 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d59bd1c6-7591-4470-b328-b0d60fb6f292-plugins-config-volume" pod "d59bd1c6-7591-4470-b328-b0d60fb6f292" (UID: "d59bd1c6-7591-4470-b328-b0d60fb6f292"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:01.262543 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.262521 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5jhw\" (UniqueName: \"kubernetes.io/projected/f0ae7b36-9635-4ea7-badb-a5a97321c841-kube-api-access-l5jhw\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:01.262647 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.262545 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d59bd1c6-7591-4470-b328-b0d60fb6f292-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:01.264311 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.264289 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59bd1c6-7591-4470-b328-b0d60fb6f292-kube-api-access-vtvm2" pod "d59bd1c6-7591-4470-b328-b0d60fb6f292" (UID: "d59bd1c6-7591-4470-b328-b0d60fb6f292"). InnerVolumeSpecName "kube-api-access-vtvm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:01.364472 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.364376 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtvm2\" (UniqueName: \"kubernetes.io/projected/d59bd1c6-7591-4470-b328-b0d60fb6f292-kube-api-access-vtvm2\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:01.672748 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.672657 4493 generic.go:356] "Generic (PLEG): container finished" podID="f0ae7b36-9635-4ea7-badb-a5a97321c841" containerID="2934f102559eb5d5832546fe5b39fdb7dbe80335dfa4281fe9dac4d845da9816" exitCode=0 Nov 03 20:58:01.672748 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.672699 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8" Nov 03 20:58:01.672748 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.672736 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8" event={"ID":"f0ae7b36-9635-4ea7-badb-a5a97321c841","Type":"ContainerDied","Data":"2934f102559eb5d5832546fe5b39fdb7dbe80335dfa4281fe9dac4d845da9816"} Nov 03 20:58:01.673304 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.672780 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8" event={"ID":"f0ae7b36-9635-4ea7-badb-a5a97321c841","Type":"ContainerDied","Data":"6f7bf6f9aaa17ce036880ef4c739fde1de7199d8bcaf1730ac944b2bca422987"} Nov 03 20:58:01.673304 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.672803 4493 scope.go:122] "RemoveContainer" containerID="2934f102559eb5d5832546fe5b39fdb7dbe80335dfa4281fe9dac4d845da9816" Nov 03 20:58:01.674406 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.674380 4493 generic.go:356] "Generic (PLEG): container finished" podID="d59bd1c6-7591-4470-b328-b0d60fb6f292" containerID="57a6a75431b24b9a0de9ca3ae3cece7137ae8a0a99dd3efe08d9746a9bb9e8d9" exitCode=0 Nov 03 20:58:01.674532 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.674418 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz" Nov 03 20:58:01.674532 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.674472 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz" event={"ID":"d59bd1c6-7591-4470-b328-b0d60fb6f292","Type":"ContainerDied","Data":"57a6a75431b24b9a0de9ca3ae3cece7137ae8a0a99dd3efe08d9746a9bb9e8d9"} Nov 03 20:58:01.674532 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.674514 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz" event={"ID":"d59bd1c6-7591-4470-b328-b0d60fb6f292","Type":"ContainerDied","Data":"c2d0225c5b47d0e91c4203588e51ab67ec13f3001bc6415a1eae227d507573bd"} Nov 03 20:58:01.683961 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.683908 4493 scope.go:122] "RemoveContainer" containerID="2934f102559eb5d5832546fe5b39fdb7dbe80335dfa4281fe9dac4d845da9816" Nov 03 20:58:01.684275 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:58:01.684249 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2934f102559eb5d5832546fe5b39fdb7dbe80335dfa4281fe9dac4d845da9816\": container with ID starting with 2934f102559eb5d5832546fe5b39fdb7dbe80335dfa4281fe9dac4d845da9816 not found: ID does not exist" containerID="2934f102559eb5d5832546fe5b39fdb7dbe80335dfa4281fe9dac4d845da9816" Nov 03 20:58:01.684404 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.684275 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2934f102559eb5d5832546fe5b39fdb7dbe80335dfa4281fe9dac4d845da9816"} err="failed to get container status \"2934f102559eb5d5832546fe5b39fdb7dbe80335dfa4281fe9dac4d845da9816\": rpc error: code = NotFound desc = could not find container \"2934f102559eb5d5832546fe5b39fdb7dbe80335dfa4281fe9dac4d845da9816\": container with ID starting with 2934f102559eb5d5832546fe5b39fdb7dbe80335dfa4281fe9dac4d845da9816 not found: ID does not exist" Nov 03 20:58:01.684404 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.684290 4493 scope.go:122] "RemoveContainer" containerID="57a6a75431b24b9a0de9ca3ae3cece7137ae8a0a99dd3efe08d9746a9bb9e8d9" Nov 03 20:58:01.693406 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.693388 4493 scope.go:122] "RemoveContainer" containerID="57a6a75431b24b9a0de9ca3ae3cece7137ae8a0a99dd3efe08d9746a9bb9e8d9" Nov 03 20:58:01.693673 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:58:01.693657 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57a6a75431b24b9a0de9ca3ae3cece7137ae8a0a99dd3efe08d9746a9bb9e8d9\": container with ID starting with 57a6a75431b24b9a0de9ca3ae3cece7137ae8a0a99dd3efe08d9746a9bb9e8d9 not found: ID does not exist" containerID="57a6a75431b24b9a0de9ca3ae3cece7137ae8a0a99dd3efe08d9746a9bb9e8d9" Nov 03 20:58:01.693710 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.693680 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57a6a75431b24b9a0de9ca3ae3cece7137ae8a0a99dd3efe08d9746a9bb9e8d9"} err="failed to get container status \"57a6a75431b24b9a0de9ca3ae3cece7137ae8a0a99dd3efe08d9746a9bb9e8d9\": rpc error: code = NotFound desc = could not find container \"57a6a75431b24b9a0de9ca3ae3cece7137ae8a0a99dd3efe08d9746a9bb9e8d9\": container with ID starting with 57a6a75431b24b9a0de9ca3ae3cece7137ae8a0a99dd3efe08d9746a9bb9e8d9 not found: ID does not exist" Nov 03 20:58:01.836916 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.835149 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz"] Nov 03 20:58:01.880906 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.880878 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cc15b1d7-6cbb65f9f5-sr6vz"] Nov 03 20:58:01.941033 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.940948 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8"] Nov 03 20:58:01.945512 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:01.945478 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cc15b1d7-58b8dcff-nw7c8"] Nov 03 20:58:02.564093 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:02.564059 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d59bd1c6-7591-4470-b328-b0d60fb6f292" path="/var/lib/kubelet/pods/d59bd1c6-7591-4470-b328-b0d60fb6f292/volumes" Nov 03 20:58:02.564628 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:02.564612 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0ae7b36-9635-4ea7-badb-a5a97321c841" path="/var/lib/kubelet/pods/f0ae7b36-9635-4ea7-badb-a5a97321c841/volumes" Nov 03 20:58:12.089028 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:12.088997 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg"] Nov 03 20:58:12.089457 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:12.089225 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" podUID="a47bd78c-2c95-4eb1-a2a3-b151a424ebdc" containerName="istio-proxy" containerID="cri-o://4aebf80031f920901e827482741b42c35b7f3cfd81268bda150b6039541523f4" gracePeriod=30 Nov 03 20:58:12.632112 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:12.632077 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7"] Nov 03 20:58:12.632306 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:12.632282 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" podUID="151eca7e-97d2-4220-a630-71e7df7d9f24" containerName="istio-proxy" containerID="cri-o://0041639d05478aa188687f291a834ccb726763dddf816154fc3d924d780708de" gracePeriod=30 Nov 03 20:58:12.643098 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:12.643065 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g"] Nov 03 20:58:12.643463 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:12.643417 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" podUID="fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb" containerName="istio-proxy" containerID="cri-o://b43663a8986a28c18666a2fdc3d0b8219cc91a0d79e30927f017f3e81b59d44f" gracePeriod=30 Nov 03 20:58:12.678830 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:12.678799 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz"] Nov 03 20:58:12.679078 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:12.679034 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" podUID="b4e6937f-8195-42d2-a7e7-d14e98aba9cf" containerName="istio-proxy" containerID="cri-o://3b7e8d4ffa54a31130a2172c8f3963780d13fbbebb28321c295b883d50e04612" gracePeriod=30 Nov 03 20:58:12.682956 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:12.682929 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7"] Nov 03 20:58:12.683153 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:12.683095 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" podUID="f62e8863-2c8c-4658-8ecf-645709ec29e2" containerName="istio-proxy" containerID="cri-o://9b2f630a7f9765da0111ba5fab6da949cad3038109aec67fad74464c5d9bddef" gracePeriod=30 Nov 03 20:58:13.761456 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:13.761416 4493 patch_prober.go:29] interesting pod/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:58:13.761939 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:13.761498 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" podUID="a47bd78c-2c95-4eb1-a2a3-b151a424ebdc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:58:14.810763 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:14.810722 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779"] Nov 03 20:58:14.811263 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:14.811005 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" podUID="156d066a-c5e2-469e-86b3-b92e319f71a0" containerName="istio-proxy" containerID="cri-o://eb8984806b24485991a8d76b1f72a5941419d1029abf0dec676d8404daac6508" gracePeriod=30 Nov 03 20:58:14.857653 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:14.857618 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm"] Nov 03 20:58:14.857913 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:14.857872 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" podUID="48c79419-fc27-4ffb-8222-63e30302cef4" containerName="istio-proxy" containerID="cri-o://0f03cd8d3f2fc52c722f700adc7a918ffdbde06257c42bb1f473f71fc5fbe371" gracePeriod=30 Nov 03 20:58:14.861990 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:14.861947 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx"] Nov 03 20:58:14.862159 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:14.862139 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" podUID="038c8d67-7cd3-48bf-a359-4ff2c577b32c" containerName="istio-proxy" containerID="cri-o://eeba465b1217eb955d9365e07d7f12c6f92682e2f4a6c26e814393a39ff78c52" gracePeriod=30 Nov 03 20:58:14.904990 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:14.904958 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc"] Nov 03 20:58:14.905190 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:14.905169 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" podUID="a3fb9508-37d2-47fa-bb02-ae7735c9f408" containerName="istio-proxy" containerID="cri-o://f7006bd03df5896a554dfa898315f2fdbbdedb9a734b66264d360ccbe4488e1d" gracePeriod=30 Nov 03 20:58:17.394203 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.394172 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:58:17.487587 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.487546 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-data\") pod \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " Nov 03 20:58:17.487787 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.487681 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-kube-api-access-h8c79\" (UniqueName: \"kubernetes.io/projected/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-kube-api-access-h8c79\") pod \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " Nov 03 20:58:17.487787 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.487720 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-token\" (UniqueName: \"kubernetes.io/projected/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-token\") pod \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " Nov 03 20:58:17.487787 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.487758 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-podinfo\") pod \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " Nov 03 20:58:17.487994 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.487857 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-data" pod "a47bd78c-2c95-4eb1-a2a3-b151a424ebdc" (UID: "a47bd78c-2c95-4eb1-a2a3-b151a424ebdc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:17.487994 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.487866 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-workload-socket\") pod \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " Nov 03 20:58:17.487994 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.487986 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-envoy\") pod \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " Nov 03 20:58:17.488116 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.488078 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-workload-socket" pod "a47bd78c-2c95-4eb1-a2a3-b151a424ebdc" (UID: "a47bd78c-2c95-4eb1-a2a3-b151a424ebdc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:17.488165 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.488109 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istiod-ca-cert\") pod \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " Nov 03 20:58:17.488222 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.488205 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-workload-certs\") pod \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " Nov 03 20:58:17.488298 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.488281 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-credential-socket\") pod \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\" (UID: \"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc\") " Nov 03 20:58:17.488565 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.488534 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istiod-ca-cert" pod "a47bd78c-2c95-4eb1-a2a3-b151a424ebdc" (UID: "a47bd78c-2c95-4eb1-a2a3-b151a424ebdc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:17.488792 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.488772 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-workload-certs" pod "a47bd78c-2c95-4eb1-a2a3-b151a424ebdc" (UID: "a47bd78c-2c95-4eb1-a2a3-b151a424ebdc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:17.489019 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.488995 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-credential-socket" pod "a47bd78c-2c95-4eb1-a2a3-b151a424ebdc" (UID: "a47bd78c-2c95-4eb1-a2a3-b151a424ebdc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:17.489651 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.489622 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:17.489749 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.489657 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:17.489749 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.489672 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:17.489749 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.489686 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:17.489749 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.489700 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:17.494840 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.494790 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-kube-api-access-h8c79" pod "a47bd78c-2c95-4eb1-a2a3-b151a424ebdc" (UID: "a47bd78c-2c95-4eb1-a2a3-b151a424ebdc"). InnerVolumeSpecName "kube-api-access-h8c79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:17.494951 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.494842 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-token" pod "a47bd78c-2c95-4eb1-a2a3-b151a424ebdc" (UID: "a47bd78c-2c95-4eb1-a2a3-b151a424ebdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:17.495065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.495033 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-envoy" pod "a47bd78c-2c95-4eb1-a2a3-b151a424ebdc" (UID: "a47bd78c-2c95-4eb1-a2a3-b151a424ebdc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:17.495164 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.495149 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-podinfo" pod "a47bd78c-2c95-4eb1-a2a3-b151a424ebdc" (UID: "a47bd78c-2c95-4eb1-a2a3-b151a424ebdc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:17.594625 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.594597 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8c79\" (UniqueName: \"kubernetes.io/projected/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-kube-api-access-h8c79\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:17.594625 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.594626 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:17.594625 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.594640 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:17.594879 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.594653 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:17.879239 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.877831 4493 generic.go:356] "Generic (PLEG): container finished" podID="a47bd78c-2c95-4eb1-a2a3-b151a424ebdc" containerID="4aebf80031f920901e827482741b42c35b7f3cfd81268bda150b6039541523f4" exitCode=0 Nov 03 20:58:17.879239 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.877928 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" event={"ID":"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc","Type":"ContainerDied","Data":"4aebf80031f920901e827482741b42c35b7f3cfd81268bda150b6039541523f4"} Nov 03 20:58:17.879239 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.877968 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" event={"ID":"a47bd78c-2c95-4eb1-a2a3-b151a424ebdc","Type":"ContainerDied","Data":"5f1414a1468a297be4492d4570febaab03b21c8d96c25e4835e9569882b2ef0a"} Nov 03 20:58:17.879239 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.877989 4493 scope.go:122] "RemoveContainer" containerID="4aebf80031f920901e827482741b42c35b7f3cfd81268bda150b6039541523f4" Nov 03 20:58:17.879239 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.878128 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg" Nov 03 20:58:17.899452 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.899420 4493 generic.go:356] "Generic (PLEG): container finished" podID="f62e8863-2c8c-4658-8ecf-645709ec29e2" containerID="9b2f630a7f9765da0111ba5fab6da949cad3038109aec67fad74464c5d9bddef" exitCode=0 Nov 03 20:58:17.899586 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.899541 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" event={"ID":"f62e8863-2c8c-4658-8ecf-645709ec29e2","Type":"ContainerDied","Data":"9b2f630a7f9765da0111ba5fab6da949cad3038109aec67fad74464c5d9bddef"} Nov 03 20:58:17.905681 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.903505 4493 generic.go:356] "Generic (PLEG): container finished" podID="b4e6937f-8195-42d2-a7e7-d14e98aba9cf" containerID="3b7e8d4ffa54a31130a2172c8f3963780d13fbbebb28321c295b883d50e04612" exitCode=0 Nov 03 20:58:17.905681 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.904534 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" event={"ID":"b4e6937f-8195-42d2-a7e7-d14e98aba9cf","Type":"ContainerDied","Data":"3b7e8d4ffa54a31130a2172c8f3963780d13fbbebb28321c295b883d50e04612"} Nov 03 20:58:17.906437 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.906099 4493 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:58:17.906437 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.906152 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" podUID="fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 20:58:17.909110 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.908963 4493 generic.go:356] "Generic (PLEG): container finished" podID="fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb" containerID="b43663a8986a28c18666a2fdc3d0b8219cc91a0d79e30927f017f3e81b59d44f" exitCode=0 Nov 03 20:58:17.909110 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.909051 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" event={"ID":"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb","Type":"ContainerDied","Data":"b43663a8986a28c18666a2fdc3d0b8219cc91a0d79e30927f017f3e81b59d44f"} Nov 03 20:58:17.917297 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.917142 4493 scope.go:122] "RemoveContainer" containerID="5d044bb04f3c6228a831e8fbf88f5adfbab9956ac9bac3af9fbf979d3ce28463" Nov 03 20:58:17.919081 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.918787 4493 generic.go:356] "Generic (PLEG): container finished" podID="151eca7e-97d2-4220-a630-71e7df7d9f24" containerID="0041639d05478aa188687f291a834ccb726763dddf816154fc3d924d780708de" exitCode=0 Nov 03 20:58:17.919081 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.918848 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" event={"ID":"151eca7e-97d2-4220-a630-71e7df7d9f24","Type":"ContainerDied","Data":"0041639d05478aa188687f291a834ccb726763dddf816154fc3d924d780708de"} Nov 03 20:58:17.930520 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.930497 4493 scope.go:122] "RemoveContainer" containerID="4aebf80031f920901e827482741b42c35b7f3cfd81268bda150b6039541523f4" Nov 03 20:58:17.930869 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:58:17.930841 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aebf80031f920901e827482741b42c35b7f3cfd81268bda150b6039541523f4\": container with ID starting with 4aebf80031f920901e827482741b42c35b7f3cfd81268bda150b6039541523f4 not found: ID does not exist" containerID="4aebf80031f920901e827482741b42c35b7f3cfd81268bda150b6039541523f4" Nov 03 20:58:17.930979 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.930961 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aebf80031f920901e827482741b42c35b7f3cfd81268bda150b6039541523f4"} err="failed to get container status \"4aebf80031f920901e827482741b42c35b7f3cfd81268bda150b6039541523f4\": rpc error: code = NotFound desc = could not find container \"4aebf80031f920901e827482741b42c35b7f3cfd81268bda150b6039541523f4\": container with ID starting with 4aebf80031f920901e827482741b42c35b7f3cfd81268bda150b6039541523f4 not found: ID does not exist" Nov 03 20:58:17.931065 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.931057 4493 scope.go:122] "RemoveContainer" containerID="5d044bb04f3c6228a831e8fbf88f5adfbab9956ac9bac3af9fbf979d3ce28463" Nov 03 20:58:17.931519 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:58:17.931500 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d044bb04f3c6228a831e8fbf88f5adfbab9956ac9bac3af9fbf979d3ce28463\": container with ID starting with 5d044bb04f3c6228a831e8fbf88f5adfbab9956ac9bac3af9fbf979d3ce28463 not found: ID does not exist" containerID="5d044bb04f3c6228a831e8fbf88f5adfbab9956ac9bac3af9fbf979d3ce28463" Nov 03 20:58:17.931618 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.931528 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d044bb04f3c6228a831e8fbf88f5adfbab9956ac9bac3af9fbf979d3ce28463"} err="failed to get container status \"5d044bb04f3c6228a831e8fbf88f5adfbab9956ac9bac3af9fbf979d3ce28463\": rpc error: code = NotFound desc = could not find container \"5d044bb04f3c6228a831e8fbf88f5adfbab9956ac9bac3af9fbf979d3ce28463\": container with ID starting with 5d044bb04f3c6228a831e8fbf88f5adfbab9956ac9bac3af9fbf979d3ce28463 not found: ID does not exist" Nov 03 20:58:17.931618 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.931554 4493 scope.go:122] "RemoveContainer" containerID="6bad799127269015d6095f4181642cbb66c6026e6391f9a53413074c11664262" Nov 03 20:58:17.951471 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:17.951396 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:58:18.003176 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.003139 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-workload-certs\") pod \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " Nov 03 20:58:18.003323 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.003217 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-data\") pod \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " Nov 03 20:58:18.003323 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.003264 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-workload-socket\") pod \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " Nov 03 20:58:18.003477 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.003422 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-podinfo\") pod \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " Nov 03 20:58:18.003601 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.003561 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-credential-socket\") pod \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " Nov 03 20:58:18.003715 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.003697 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-kube-api-access-tgc5w\" (UniqueName: \"kubernetes.io/projected/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-kube-api-access-tgc5w\") pod \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " Nov 03 20:58:18.003789 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.003776 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-token\" (UniqueName: \"kubernetes.io/projected/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-token\") pod \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " Nov 03 20:58:18.003874 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.003846 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-envoy\") pod \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " Nov 03 20:58:18.003874 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.003868 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istiod-ca-cert\") pod \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\" (UID: \"b4e6937f-8195-42d2-a7e7-d14e98aba9cf\") " Nov 03 20:58:18.004678 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.004653 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istiod-ca-cert" pod "b4e6937f-8195-42d2-a7e7-d14e98aba9cf" (UID: "b4e6937f-8195-42d2-a7e7-d14e98aba9cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:18.004868 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.004852 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-workload-certs" pod "b4e6937f-8195-42d2-a7e7-d14e98aba9cf" (UID: "b4e6937f-8195-42d2-a7e7-d14e98aba9cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:18.005005 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.004973 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-data" pod "b4e6937f-8195-42d2-a7e7-d14e98aba9cf" (UID: "b4e6937f-8195-42d2-a7e7-d14e98aba9cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:18.005097 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.005080 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-workload-socket" pod "b4e6937f-8195-42d2-a7e7-d14e98aba9cf" (UID: "b4e6937f-8195-42d2-a7e7-d14e98aba9cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:18.012378 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.008932 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-credential-socket" pod "b4e6937f-8195-42d2-a7e7-d14e98aba9cf" (UID: "b4e6937f-8195-42d2-a7e7-d14e98aba9cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:18.012378 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.012040 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-envoy" pod "b4e6937f-8195-42d2-a7e7-d14e98aba9cf" (UID: "b4e6937f-8195-42d2-a7e7-d14e98aba9cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:18.022823 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.021709 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-kube-api-access-tgc5w" pod "b4e6937f-8195-42d2-a7e7-d14e98aba9cf" (UID: "b4e6937f-8195-42d2-a7e7-d14e98aba9cf"). InnerVolumeSpecName "kube-api-access-tgc5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:18.022823 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.021884 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-podinfo" pod "b4e6937f-8195-42d2-a7e7-d14e98aba9cf" (UID: "b4e6937f-8195-42d2-a7e7-d14e98aba9cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:18.024992 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.024964 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-token" pod "b4e6937f-8195-42d2-a7e7-d14e98aba9cf" (UID: "b4e6937f-8195-42d2-a7e7-d14e98aba9cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:18.040166 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.040142 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:58:18.051879 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.051851 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg"] Nov 03 20:58:18.061504 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.061480 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0e8374c4-openshift-default-578f7dd545-f7rmg"] Nov 03 20:58:18.080252 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.080208 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:58:18.084736 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.084719 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:58:18.105687 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.105662 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-workload-certs\") pod \"151eca7e-97d2-4220-a630-71e7df7d9f24\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " Nov 03 20:58:18.105780 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.105718 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-envoy\") pod \"f62e8863-2c8c-4658-8ecf-645709ec29e2\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " Nov 03 20:58:18.105780 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.105754 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-data\") pod \"151eca7e-97d2-4220-a630-71e7df7d9f24\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " Nov 03 20:58:18.105874 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.105783 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-envoy\") pod \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " Nov 03 20:58:18.105874 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.105813 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-data\") pod \"f62e8863-2c8c-4658-8ecf-645709ec29e2\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " Nov 03 20:58:18.105874 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.105849 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-podinfo\") pod \"151eca7e-97d2-4220-a630-71e7df7d9f24\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " Nov 03 20:58:18.106017 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.105929 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istiod-ca-cert\") pod \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " Nov 03 20:58:18.106017 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.105956 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-workload-certs\") pod \"f62e8863-2c8c-4658-8ecf-645709ec29e2\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " Nov 03 20:58:18.106017 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.105963 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-workload-certs" pod "151eca7e-97d2-4220-a630-71e7df7d9f24" (UID: "151eca7e-97d2-4220-a630-71e7df7d9f24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:18.106017 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.105993 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-credential-socket\") pod \"f62e8863-2c8c-4658-8ecf-645709ec29e2\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " Nov 03 20:58:18.106207 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.106035 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-data" pod "151eca7e-97d2-4220-a630-71e7df7d9f24" (UID: "151eca7e-97d2-4220-a630-71e7df7d9f24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:18.106207 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.106058 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f62e8863-2c8c-4658-8ecf-645709ec29e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f62e8863-2c8c-4658-8ecf-645709ec29e2-istiod-ca-cert\") pod \"f62e8863-2c8c-4658-8ecf-645709ec29e2\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " Nov 03 20:58:18.106207 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.106102 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-data\") pod \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " Nov 03 20:58:18.106207 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.106137 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f62e8863-2c8c-4658-8ecf-645709ec29e2-kube-api-access-ndb6g\" (UniqueName: \"kubernetes.io/projected/f62e8863-2c8c-4658-8ecf-645709ec29e2-kube-api-access-ndb6g\") pod \"f62e8863-2c8c-4658-8ecf-645709ec29e2\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " Nov 03 20:58:18.106410 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.106204 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-credential-socket" pod "f62e8863-2c8c-4658-8ecf-645709ec29e2" (UID: "f62e8863-2c8c-4658-8ecf-645709ec29e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:18.106410 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.106229 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-kube-api-access-2cbx8\" (UniqueName: \"kubernetes.io/projected/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-kube-api-access-2cbx8\") pod \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " Nov 03 20:58:18.106410 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.106271 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151eca7e-97d2-4220-a630-71e7df7d9f24-kube-api-access-6vf25\" (UniqueName: \"kubernetes.io/projected/151eca7e-97d2-4220-a630-71e7df7d9f24-kube-api-access-6vf25\") pod \"151eca7e-97d2-4220-a630-71e7df7d9f24\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " Nov 03 20:58:18.106410 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.106380 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-podinfo\") pod \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " Nov 03 20:58:18.106613 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.106425 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-workload-socket\") pod \"151eca7e-97d2-4220-a630-71e7df7d9f24\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " Nov 03 20:58:18.106613 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.106499 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-workload-socket\") pod \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " Nov 03 20:58:18.106613 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.106532 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istiod-ca-cert" pod "fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb" (UID: "fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:18.106613 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.106551 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-workload-certs\") pod \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " Nov 03 20:58:18.106613 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.106604 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-credential-socket\") pod \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " Nov 03 20:58:18.106857 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.106662 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-podinfo\") pod \"f62e8863-2c8c-4658-8ecf-645709ec29e2\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " Nov 03 20:58:18.106857 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.106730 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-credential-socket\") pod \"151eca7e-97d2-4220-a630-71e7df7d9f24\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " Nov 03 20:58:18.106857 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.106748 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-workload-certs" pod "f62e8863-2c8c-4658-8ecf-645709ec29e2" (UID: "f62e8863-2c8c-4658-8ecf-645709ec29e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:18.106857 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.106774 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-token\" (UniqueName: \"kubernetes.io/projected/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-token\") pod \"151eca7e-97d2-4220-a630-71e7df7d9f24\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " Nov 03 20:58:18.106857 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.106819 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-envoy\") pod \"151eca7e-97d2-4220-a630-71e7df7d9f24\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " Nov 03 20:58:18.107101 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.106925 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-token\" (UniqueName: \"kubernetes.io/projected/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-token\") pod \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\" (UID: \"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb\") " Nov 03 20:58:18.107101 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.106958 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/151eca7e-97d2-4220-a630-71e7df7d9f24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151eca7e-97d2-4220-a630-71e7df7d9f24-istiod-ca-cert\") pod \"151eca7e-97d2-4220-a630-71e7df7d9f24\" (UID: \"151eca7e-97d2-4220-a630-71e7df7d9f24\") " Nov 03 20:58:18.107101 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.106990 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-token\" (UniqueName: \"kubernetes.io/projected/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-token\") pod \"f62e8863-2c8c-4658-8ecf-645709ec29e2\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " Nov 03 20:58:18.107101 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.107044 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-workload-socket\") pod \"f62e8863-2c8c-4658-8ecf-645709ec29e2\" (UID: \"f62e8863-2c8c-4658-8ecf-645709ec29e2\") " Nov 03 20:58:18.107101 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.107051 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-credential-socket" pod "fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb" (UID: "fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:18.107368 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.107243 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-workload-socket" pod "151eca7e-97d2-4220-a630-71e7df7d9f24" (UID: "151eca7e-97d2-4220-a630-71e7df7d9f24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:18.108033 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.107739 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-workload-socket" pod "fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb" (UID: "fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:18.108033 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.107692 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-data" pod "f62e8863-2c8c-4658-8ecf-645709ec29e2" (UID: "f62e8863-2c8c-4658-8ecf-645709ec29e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:18.108033 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.107964 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-workload-socket" pod "f62e8863-2c8c-4658-8ecf-645709ec29e2" (UID: "f62e8863-2c8c-4658-8ecf-645709ec29e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:18.108239 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.108078 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-credential-socket" pod "151eca7e-97d2-4220-a630-71e7df7d9f24" (UID: "151eca7e-97d2-4220-a630-71e7df7d9f24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:18.108931 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.108873 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-podinfo" pod "151eca7e-97d2-4220-a630-71e7df7d9f24" (UID: "151eca7e-97d2-4220-a630-71e7df7d9f24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:18.109019 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.108946 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-workload-certs" pod "fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb" (UID: "fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:18.109163 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.109141 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.109268 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.109166 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.109268 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.109180 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.109268 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.109193 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.109268 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.109204 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.109268 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.109217 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.109268 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.109230 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.109268 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.109244 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.109268 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.109257 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.109741 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.109270 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.109741 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.109286 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgc5w\" (UniqueName: \"kubernetes.io/projected/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-kube-api-access-tgc5w\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.109741 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.109301 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.109741 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.109316 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.109741 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.109367 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.109741 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.109382 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.109741 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.109395 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.109741 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.109408 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.109741 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.109422 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.109741 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.109436 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.109741 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.109450 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.109741 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.109464 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e6937f-8195-42d2-a7e7-d14e98aba9cf-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.110305 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.109898 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-envoy" pod "fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb" (UID: "fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:18.110305 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.110171 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-envoy" pod "f62e8863-2c8c-4658-8ecf-645709ec29e2" (UID: "f62e8863-2c8c-4658-8ecf-645709ec29e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:18.110672 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.110643 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-podinfo" pod "fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb" (UID: "fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:18.110918 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.110717 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-token" pod "fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb" (UID: "fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:18.110918 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.110861 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-envoy" pod "151eca7e-97d2-4220-a630-71e7df7d9f24" (UID: "151eca7e-97d2-4220-a630-71e7df7d9f24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:18.111069 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.111046 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/151eca7e-97d2-4220-a630-71e7df7d9f24-istiod-ca-cert" pod "151eca7e-97d2-4220-a630-71e7df7d9f24" (UID: "151eca7e-97d2-4220-a630-71e7df7d9f24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:18.111231 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.111212 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f62e8863-2c8c-4658-8ecf-645709ec29e2-istiod-ca-cert" pod "f62e8863-2c8c-4658-8ecf-645709ec29e2" (UID: "f62e8863-2c8c-4658-8ecf-645709ec29e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:18.111398 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.111374 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-data" pod "fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb" (UID: "fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:18.111707 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.111669 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-podinfo" pod "f62e8863-2c8c-4658-8ecf-645709ec29e2" (UID: "f62e8863-2c8c-4658-8ecf-645709ec29e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:18.112003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.111979 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f62e8863-2c8c-4658-8ecf-645709ec29e2-kube-api-access-ndb6g" pod "f62e8863-2c8c-4658-8ecf-645709ec29e2" (UID: "f62e8863-2c8c-4658-8ecf-645709ec29e2"). InnerVolumeSpecName "kube-api-access-ndb6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:18.112675 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.112652 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-kube-api-access-2cbx8" pod "fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb" (UID: "fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb"). InnerVolumeSpecName "kube-api-access-2cbx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:18.113195 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.113174 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-token" pod "f62e8863-2c8c-4658-8ecf-645709ec29e2" (UID: "f62e8863-2c8c-4658-8ecf-645709ec29e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:18.113617 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.113601 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-token" pod "151eca7e-97d2-4220-a630-71e7df7d9f24" (UID: "151eca7e-97d2-4220-a630-71e7df7d9f24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:18.114134 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.114117 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151eca7e-97d2-4220-a630-71e7df7d9f24-kube-api-access-6vf25" pod "151eca7e-97d2-4220-a630-71e7df7d9f24" (UID: "151eca7e-97d2-4220-a630-71e7df7d9f24"). InnerVolumeSpecName "kube-api-access-6vf25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:18.211261 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.211178 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.211261 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.211214 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151eca7e-97d2-4220-a630-71e7df7d9f24-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.211261 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.211231 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.211261 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.211247 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.211261 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.211263 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.211579 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.211276 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.211579 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.211290 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f62e8863-2c8c-4658-8ecf-645709ec29e2-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.211579 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.211302 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.211579 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.211317 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndb6g\" (UniqueName: \"kubernetes.io/projected/f62e8863-2c8c-4658-8ecf-645709ec29e2-kube-api-access-ndb6g\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.211579 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.211373 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cbx8\" (UniqueName: \"kubernetes.io/projected/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-kube-api-access-2cbx8\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.211579 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.211388 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vf25\" (UniqueName: \"kubernetes.io/projected/151eca7e-97d2-4220-a630-71e7df7d9f24-kube-api-access-6vf25\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.211579 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.211401 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.211579 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.211415 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f62e8863-2c8c-4658-8ecf-645709ec29e2-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.211579 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.211430 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.211579 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.211443 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151eca7e-97d2-4220-a630-71e7df7d9f24-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:18.570954 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.570923 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a47bd78c-2c95-4eb1-a2a3-b151a424ebdc" path="/var/lib/kubelet/pods/a47bd78c-2c95-4eb1-a2a3-b151a424ebdc/volumes" Nov 03 20:58:18.928649 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.928558 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" Nov 03 20:58:18.928818 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.928557 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7" event={"ID":"f62e8863-2c8c-4658-8ecf-645709ec29e2","Type":"ContainerDied","Data":"9cdbd3fa0a52884de72616a21a12160347190d46d77591c7a32a94f76a423118"} Nov 03 20:58:18.928818 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.928723 4493 scope.go:122] "RemoveContainer" containerID="9b2f630a7f9765da0111ba5fab6da949cad3038109aec67fad74464c5d9bddef" Nov 03 20:58:18.930216 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.930195 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" Nov 03 20:58:18.930467 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.930212 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz" event={"ID":"b4e6937f-8195-42d2-a7e7-d14e98aba9cf","Type":"ContainerDied","Data":"320a3ce5632e56254d522ad16a3d6cf1958a7c4bfcb0a58efbd3bb62efbb58af"} Nov 03 20:58:18.931904 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.931878 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" Nov 03 20:58:18.932009 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.931890 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g" event={"ID":"fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb","Type":"ContainerDied","Data":"6ba62f3a0b3ccab39ac47a92a1e86a4e1b665797a1598a81f441163c92344849"} Nov 03 20:58:18.934022 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.934001 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" event={"ID":"151eca7e-97d2-4220-a630-71e7df7d9f24","Type":"ContainerDied","Data":"8cb29f8f6b24fd1190bc6eb0f32d603b88768b16e7f11105f0762c9cdebfd1ab"} Nov 03 20:58:18.934124 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.934100 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7" Nov 03 20:58:18.943693 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.943673 4493 scope.go:122] "RemoveContainer" containerID="3b7e8d4ffa54a31130a2172c8f3963780d13fbbebb28321c295b883d50e04612" Nov 03 20:58:18.954048 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.954023 4493 scope.go:122] "RemoveContainer" containerID="b43663a8986a28c18666a2fdc3d0b8219cc91a0d79e30927f017f3e81b59d44f" Nov 03 20:58:18.965021 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:18.965001 4493 scope.go:122] "RemoveContainer" containerID="0041639d05478aa188687f291a834ccb726763dddf816154fc3d924d780708de" Nov 03 20:58:19.086168 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:19.086137 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7"] Nov 03 20:58:19.132771 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:19.132742 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-z66t7"] Nov 03 20:58:19.237579 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:19.237482 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g"] Nov 03 20:58:19.241459 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:19.241435 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-w9x2g"] Nov 03 20:58:19.347257 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:19.347221 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7"] Nov 03 20:58:19.347477 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:19.347328 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xc5z7"] Nov 03 20:58:19.449441 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:19.449410 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz"] Nov 03 20:58:19.453087 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:19.453065 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-xkgxz"] Nov 03 20:58:19.507584 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:19.507508 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:58:19.507584 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:19.507562 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" podUID="156d066a-c5e2-469e-86b3-b92e319f71a0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:58:19.946199 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:19.946144 4493 generic.go:356] "Generic (PLEG): container finished" podID="156d066a-c5e2-469e-86b3-b92e319f71a0" containerID="eb8984806b24485991a8d76b1f72a5941419d1029abf0dec676d8404daac6508" exitCode=0 Nov 03 20:58:19.946671 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:19.946222 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" event={"ID":"156d066a-c5e2-469e-86b3-b92e319f71a0","Type":"ContainerDied","Data":"eb8984806b24485991a8d76b1f72a5941419d1029abf0dec676d8404daac6508"} Nov 03 20:58:19.971252 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:19.971228 4493 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 20:58:19.971373 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:19.971277 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" podUID="48c79419-fc27-4ffb-8222-63e30302cef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 20:58:20.159153 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.159120 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:58:20.242347 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.240810 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-credential-socket\") pod \"156d066a-c5e2-469e-86b3-b92e319f71a0\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " Nov 03 20:58:20.242347 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.240923 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-podinfo\") pod \"156d066a-c5e2-469e-86b3-b92e319f71a0\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " Nov 03 20:58:20.242347 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.240959 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-workload-socket\") pod \"156d066a-c5e2-469e-86b3-b92e319f71a0\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " Nov 03 20:58:20.242347 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.241108 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-data\") pod \"156d066a-c5e2-469e-86b3-b92e319f71a0\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " Nov 03 20:58:20.242347 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.241164 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-workload-certs\") pod \"156d066a-c5e2-469e-86b3-b92e319f71a0\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " Nov 03 20:58:20.242347 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.241399 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-data" pod "156d066a-c5e2-469e-86b3-b92e319f71a0" (UID: "156d066a-c5e2-469e-86b3-b92e319f71a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:20.242347 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.241471 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-workload-certs" pod "156d066a-c5e2-469e-86b3-b92e319f71a0" (UID: "156d066a-c5e2-469e-86b3-b92e319f71a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:20.242347 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.241540 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-envoy\") pod \"156d066a-c5e2-469e-86b3-b92e319f71a0\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " Nov 03 20:58:20.242347 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.241611 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-workload-socket" pod "156d066a-c5e2-469e-86b3-b92e319f71a0" (UID: "156d066a-c5e2-469e-86b3-b92e319f71a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:20.242347 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.241909 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-token\" (UniqueName: \"kubernetes.io/projected/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-token\") pod \"156d066a-c5e2-469e-86b3-b92e319f71a0\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " Nov 03 20:58:20.242347 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.241992 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/156d066a-c5e2-469e-86b3-b92e319f71a0-kube-api-access-qtrzr\" (UniqueName: \"kubernetes.io/projected/156d066a-c5e2-469e-86b3-b92e319f71a0-kube-api-access-qtrzr\") pod \"156d066a-c5e2-469e-86b3-b92e319f71a0\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " Nov 03 20:58:20.242347 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.241908 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-credential-socket" pod "156d066a-c5e2-469e-86b3-b92e319f71a0" (UID: "156d066a-c5e2-469e-86b3-b92e319f71a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:20.243049 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.242091 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/156d066a-c5e2-469e-86b3-b92e319f71a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/156d066a-c5e2-469e-86b3-b92e319f71a0-istiod-ca-cert\") pod \"156d066a-c5e2-469e-86b3-b92e319f71a0\" (UID: \"156d066a-c5e2-469e-86b3-b92e319f71a0\") " Nov 03 20:58:20.246377 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.242799 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/156d066a-c5e2-469e-86b3-b92e319f71a0-istiod-ca-cert" pod "156d066a-c5e2-469e-86b3-b92e319f71a0" (UID: "156d066a-c5e2-469e-86b3-b92e319f71a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:20.246377 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.244231 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/156d066a-c5e2-469e-86b3-b92e319f71a0-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.246377 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.244253 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.246377 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.244267 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.246377 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.244278 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.246377 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.244289 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.251190 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.251167 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-token" pod "156d066a-c5e2-469e-86b3-b92e319f71a0" (UID: "156d066a-c5e2-469e-86b3-b92e319f71a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:20.252562 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.252499 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-envoy" pod "156d066a-c5e2-469e-86b3-b92e319f71a0" (UID: "156d066a-c5e2-469e-86b3-b92e319f71a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:20.253051 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.253031 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-podinfo" pod "156d066a-c5e2-469e-86b3-b92e319f71a0" (UID: "156d066a-c5e2-469e-86b3-b92e319f71a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:20.253537 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.253512 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156d066a-c5e2-469e-86b3-b92e319f71a0-kube-api-access-qtrzr" pod "156d066a-c5e2-469e-86b3-b92e319f71a0" (UID: "156d066a-c5e2-469e-86b3-b92e319f71a0"). InnerVolumeSpecName "kube-api-access-qtrzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:20.260994 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.260975 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:58:20.298734 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.298705 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:58:20.308243 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.308215 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:58:20.345642 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.345613 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-envoy\") pod \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " Nov 03 20:58:20.345760 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.345647 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-credential-socket\") pod \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " Nov 03 20:58:20.345760 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.345671 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-data\") pod \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " Nov 03 20:58:20.345760 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.345694 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-podinfo\") pod \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " Nov 03 20:58:20.345900 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.345759 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-workload-socket\") pod \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " Nov 03 20:58:20.345997 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.345974 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-credential-socket" pod "a3fb9508-37d2-47fa-bb02-ae7735c9f408" (UID: "a3fb9508-37d2-47fa-bb02-ae7735c9f408"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:20.345997 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.345978 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istiod-ca-cert\") pod \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " Nov 03 20:58:20.346102 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.345954 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-data" pod "038c8d67-7cd3-48bf-a359-4ff2c577b32c" (UID: "038c8d67-7cd3-48bf-a359-4ff2c577b32c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:20.346102 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.346026 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-workload-socket" pod "a3fb9508-37d2-47fa-bb02-ae7735c9f408" (UID: "a3fb9508-37d2-47fa-bb02-ae7735c9f408"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:20.346102 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.346053 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48c79419-fc27-4ffb-8222-63e30302cef4-istio-token\" (UniqueName: \"kubernetes.io/projected/48c79419-fc27-4ffb-8222-63e30302cef4-istio-token\") pod \"48c79419-fc27-4ffb-8222-63e30302cef4\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " Nov 03 20:58:20.346102 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.346086 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-credential-socket\") pod \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " Nov 03 20:58:20.346288 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.346132 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-istio-data\") pod \"48c79419-fc27-4ffb-8222-63e30302cef4\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " Nov 03 20:58:20.346288 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.346175 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-token\" (UniqueName: \"kubernetes.io/projected/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-token\") pod \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " Nov 03 20:58:20.346288 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.346211 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-workload-socket\") pod \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " Nov 03 20:58:20.346288 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.346242 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-workload-certs\") pod \"48c79419-fc27-4ffb-8222-63e30302cef4\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " Nov 03 20:58:20.346288 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.346278 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istiod-ca-cert" pod "a3fb9508-37d2-47fa-bb02-ae7735c9f408" (UID: "a3fb9508-37d2-47fa-bb02-ae7735c9f408"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:20.346602 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.346280 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48c79419-fc27-4ffb-8222-63e30302cef4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c79419-fc27-4ffb-8222-63e30302cef4-istio-podinfo\") pod \"48c79419-fc27-4ffb-8222-63e30302cef4\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " Nov 03 20:58:20.346747 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.346523 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48c79419-fc27-4ffb-8222-63e30302cef4-kube-api-access-kzxgm\" (UniqueName: \"kubernetes.io/projected/48c79419-fc27-4ffb-8222-63e30302cef4-kube-api-access-kzxgm\") pod \"48c79419-fc27-4ffb-8222-63e30302cef4\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " Nov 03 20:58:20.346875 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.346395 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-credential-socket" pod "038c8d67-7cd3-48bf-a359-4ff2c577b32c" (UID: "038c8d67-7cd3-48bf-a359-4ff2c577b32c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:20.346875 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.346701 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-istio-data" pod "48c79419-fc27-4ffb-8222-63e30302cef4" (UID: "48c79419-fc27-4ffb-8222-63e30302cef4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:20.347079 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.346890 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-workload-certs" pod "48c79419-fc27-4ffb-8222-63e30302cef4" (UID: "48c79419-fc27-4ffb-8222-63e30302cef4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:20.347079 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.346938 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-workload-certs\") pod \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " Nov 03 20:58:20.347195 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.347118 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-workload-socket" pod "038c8d67-7cd3-48bf-a359-4ff2c577b32c" (UID: "038c8d67-7cd3-48bf-a359-4ff2c577b32c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:20.347195 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.347157 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-workload-certs" pod "038c8d67-7cd3-48bf-a359-4ff2c577b32c" (UID: "038c8d67-7cd3-48bf-a359-4ff2c577b32c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:20.347298 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.347240 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-credential-socket\") pod \"48c79419-fc27-4ffb-8222-63e30302cef4\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " Nov 03 20:58:20.347298 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.347270 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istiod-ca-cert\") pod \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " Nov 03 20:58:20.347724 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.347304 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-workload-certs\") pod \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " Nov 03 20:58:20.347724 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.347567 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-credential-socket" pod "48c79419-fc27-4ffb-8222-63e30302cef4" (UID: "48c79419-fc27-4ffb-8222-63e30302cef4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:20.347724 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.347709 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-workload-certs" pod "a3fb9508-37d2-47fa-bb02-ae7735c9f408" (UID: "a3fb9508-37d2-47fa-bb02-ae7735c9f408"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:20.347884 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.347735 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istiod-ca-cert" pod "038c8d67-7cd3-48bf-a359-4ff2c577b32c" (UID: "038c8d67-7cd3-48bf-a359-4ff2c577b32c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:20.348000 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.347982 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-workload-socket\") pod \"48c79419-fc27-4ffb-8222-63e30302cef4\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " Nov 03 20:58:20.348096 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.348078 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-istio-envoy\") pod \"48c79419-fc27-4ffb-8222-63e30302cef4\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " Nov 03 20:58:20.348158 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.348004 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-workload-socket" pod "48c79419-fc27-4ffb-8222-63e30302cef4" (UID: "48c79419-fc27-4ffb-8222-63e30302cef4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:20.348218 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.348162 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48c79419-fc27-4ffb-8222-63e30302cef4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c79419-fc27-4ffb-8222-63e30302cef4-istiod-ca-cert\") pod \"48c79419-fc27-4ffb-8222-63e30302cef4\" (UID: \"48c79419-fc27-4ffb-8222-63e30302cef4\") " Nov 03 20:58:20.348448 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.348426 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-token\" (UniqueName: \"kubernetes.io/projected/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-token\") pod \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " Nov 03 20:58:20.348558 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.348489 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-envoy\") pod \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " Nov 03 20:58:20.348558 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.348541 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/038c8d67-7cd3-48bf-a359-4ff2c577b32c-kube-api-access-ldhtf\" (UniqueName: \"kubernetes.io/projected/038c8d67-7cd3-48bf-a359-4ff2c577b32c-kube-api-access-ldhtf\") pod \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\" (UID: \"038c8d67-7cd3-48bf-a359-4ff2c577b32c\") " Nov 03 20:58:20.348728 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.348571 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3fb9508-37d2-47fa-bb02-ae7735c9f408-kube-api-access-px8jt\" (UniqueName: \"kubernetes.io/projected/a3fb9508-37d2-47fa-bb02-ae7735c9f408-kube-api-access-px8jt\") pod \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " Nov 03 20:58:20.348728 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.348623 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-podinfo\") pod \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " Nov 03 20:58:20.348728 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.348664 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c79419-fc27-4ffb-8222-63e30302cef4-istiod-ca-cert" pod "48c79419-fc27-4ffb-8222-63e30302cef4" (UID: "48c79419-fc27-4ffb-8222-63e30302cef4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:20.348900 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.348790 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c79419-fc27-4ffb-8222-63e30302cef4-istio-token" pod "48c79419-fc27-4ffb-8222-63e30302cef4" (UID: "48c79419-fc27-4ffb-8222-63e30302cef4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:20.349036 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.349012 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-data\") pod \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\" (UID: \"a3fb9508-37d2-47fa-bb02-ae7735c9f408\") " Nov 03 20:58:20.349199 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.349176 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-envoy" pod "038c8d67-7cd3-48bf-a359-4ff2c577b32c" (UID: "038c8d67-7cd3-48bf-a359-4ff2c577b32c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:20.349320 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.349294 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-data" pod "a3fb9508-37d2-47fa-bb02-ae7735c9f408" (UID: "a3fb9508-37d2-47fa-bb02-ae7735c9f408"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:20.349490 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.349465 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48c79419-fc27-4ffb-8222-63e30302cef4-istio-podinfo" pod "48c79419-fc27-4ffb-8222-63e30302cef4" (UID: "48c79419-fc27-4ffb-8222-63e30302cef4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:20.350049 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.350025 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.350130 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.350057 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.350130 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.350072 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.350130 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.350087 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.350130 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.350101 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.350130 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.350115 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.350130 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.350130 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.350607 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.350145 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c79419-fc27-4ffb-8222-63e30302cef4-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.350607 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.350164 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.350607 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.350179 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.350607 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.350193 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.350607 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.350206 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.350607 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.350220 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c79419-fc27-4ffb-8222-63e30302cef4-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.350607 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.350234 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/038c8d67-7cd3-48bf-a359-4ff2c577b32c-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.350607 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.350247 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.350607 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.350261 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.350607 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.350275 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.350607 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.350288 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.350607 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.350303 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c79419-fc27-4ffb-8222-63e30302cef4-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.350607 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.350317 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.350607 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.350352 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/156d066a-c5e2-469e-86b3-b92e319f71a0-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.350607 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.350366 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtrzr\" (UniqueName: \"kubernetes.io/projected/156d066a-c5e2-469e-86b3-b92e319f71a0-kube-api-access-qtrzr\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.350607 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.350467 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-podinfo" pod "038c8d67-7cd3-48bf-a359-4ff2c577b32c" (UID: "038c8d67-7cd3-48bf-a359-4ff2c577b32c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:20.351474 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.350664 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c79419-fc27-4ffb-8222-63e30302cef4-kube-api-access-kzxgm" pod "48c79419-fc27-4ffb-8222-63e30302cef4" (UID: "48c79419-fc27-4ffb-8222-63e30302cef4"). InnerVolumeSpecName "kube-api-access-kzxgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:20.351474 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.351260 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-token" pod "038c8d67-7cd3-48bf-a359-4ff2c577b32c" (UID: "038c8d67-7cd3-48bf-a359-4ff2c577b32c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:20.351474 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.351427 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-token" pod "a3fb9508-37d2-47fa-bb02-ae7735c9f408" (UID: "a3fb9508-37d2-47fa-bb02-ae7735c9f408"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:20.351666 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.351596 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-podinfo" pod "a3fb9508-37d2-47fa-bb02-ae7735c9f408" (UID: "a3fb9508-37d2-47fa-bb02-ae7735c9f408"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:20.352005 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.351981 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-istio-envoy" pod "48c79419-fc27-4ffb-8222-63e30302cef4" (UID: "48c79419-fc27-4ffb-8222-63e30302cef4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:20.352005 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.351994 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/038c8d67-7cd3-48bf-a359-4ff2c577b32c-kube-api-access-ldhtf" pod "038c8d67-7cd3-48bf-a359-4ff2c577b32c" (UID: "038c8d67-7cd3-48bf-a359-4ff2c577b32c"). InnerVolumeSpecName "kube-api-access-ldhtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:20.352238 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.352222 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3fb9508-37d2-47fa-bb02-ae7735c9f408-kube-api-access-px8jt" pod "a3fb9508-37d2-47fa-bb02-ae7735c9f408" (UID: "a3fb9508-37d2-47fa-bb02-ae7735c9f408"). InnerVolumeSpecName "kube-api-access-px8jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:20.352729 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.352713 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-envoy" pod "a3fb9508-37d2-47fa-bb02-ae7735c9f408" (UID: "a3fb9508-37d2-47fa-bb02-ae7735c9f408"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:20.451732 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.451650 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.451732 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.451678 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/038c8d67-7cd3-48bf-a359-4ff2c577b32c-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.451732 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.451687 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzxgm\" (UniqueName: \"kubernetes.io/projected/48c79419-fc27-4ffb-8222-63e30302cef4-kube-api-access-kzxgm\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.451732 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.451697 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c79419-fc27-4ffb-8222-63e30302cef4-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.451732 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.451705 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.451732 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.451715 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.451732 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.451723 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldhtf\" (UniqueName: \"kubernetes.io/projected/038c8d67-7cd3-48bf-a359-4ff2c577b32c-kube-api-access-ldhtf\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.451732 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.451731 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px8jt\" (UniqueName: \"kubernetes.io/projected/a3fb9508-37d2-47fa-bb02-ae7735c9f408-kube-api-access-px8jt\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.452056 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.451741 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3fb9508-37d2-47fa-bb02-ae7735c9f408-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:20.564958 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.564926 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="151eca7e-97d2-4220-a630-71e7df7d9f24" path="/var/lib/kubelet/pods/151eca7e-97d2-4220-a630-71e7df7d9f24/volumes" Nov 03 20:58:20.565487 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.565470 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4e6937f-8195-42d2-a7e7-d14e98aba9cf" path="/var/lib/kubelet/pods/b4e6937f-8195-42d2-a7e7-d14e98aba9cf/volumes" Nov 03 20:58:20.565923 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.565909 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f62e8863-2c8c-4658-8ecf-645709ec29e2" path="/var/lib/kubelet/pods/f62e8863-2c8c-4658-8ecf-645709ec29e2/volumes" Nov 03 20:58:20.566747 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.566731 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb" path="/var/lib/kubelet/pods/fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb/volumes" Nov 03 20:58:20.644614 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:58:20.644576 4493 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod038c8d67_7cd3_48bf_a359_4ff2c577b32c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c79419_fc27_4ffb_8222_63e30302cef4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod038c8d67_7cd3_48bf_a359_4ff2c577b32c.slice/crio-ec9fc930b223318c293c2b3f57405e153b139a4c8e0df2b83dcbc08a817c24c8\": RecentStats: unable to find data in memory cache]" Nov 03 20:58:20.957918 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.957866 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" event={"ID":"156d066a-c5e2-469e-86b3-b92e319f71a0","Type":"ContainerDied","Data":"c216ecb47206096641e4b55b82d05b98e1969eb92383325f24dda6dcf34b3aee"} Nov 03 20:58:20.958496 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.957956 4493 scope.go:122] "RemoveContainer" containerID="eb8984806b24485991a8d76b1f72a5941419d1029abf0dec676d8404daac6508" Nov 03 20:58:20.958496 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.957893 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779" Nov 03 20:58:20.961477 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.961451 4493 generic.go:356] "Generic (PLEG): container finished" podID="48c79419-fc27-4ffb-8222-63e30302cef4" containerID="0f03cd8d3f2fc52c722f700adc7a918ffdbde06257c42bb1f473f71fc5fbe371" exitCode=0 Nov 03 20:58:20.961568 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.961501 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" Nov 03 20:58:20.961646 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.961626 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" event={"ID":"48c79419-fc27-4ffb-8222-63e30302cef4","Type":"ContainerDied","Data":"0f03cd8d3f2fc52c722f700adc7a918ffdbde06257c42bb1f473f71fc5fbe371"} Nov 03 20:58:20.961731 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.961668 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm" event={"ID":"48c79419-fc27-4ffb-8222-63e30302cef4","Type":"ContainerDied","Data":"b751a7580e9575d84f2504e77170ebc397ed6cb43410746be56916f0416550db"} Nov 03 20:58:20.964156 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.964131 4493 generic.go:356] "Generic (PLEG): container finished" podID="a3fb9508-37d2-47fa-bb02-ae7735c9f408" containerID="f7006bd03df5896a554dfa898315f2fdbbdedb9a734b66264d360ccbe4488e1d" exitCode=0 Nov 03 20:58:20.964288 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.964198 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" Nov 03 20:58:20.964288 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.964232 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" event={"ID":"a3fb9508-37d2-47fa-bb02-ae7735c9f408","Type":"ContainerDied","Data":"f7006bd03df5896a554dfa898315f2fdbbdedb9a734b66264d360ccbe4488e1d"} Nov 03 20:58:20.964488 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.964288 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc" event={"ID":"a3fb9508-37d2-47fa-bb02-ae7735c9f408","Type":"ContainerDied","Data":"a6d094f20c307d68add188d4b6745cd6e2ed6f908d6fc60c784443dff6c00f72"} Nov 03 20:58:20.965982 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.965956 4493 generic.go:356] "Generic (PLEG): container finished" podID="038c8d67-7cd3-48bf-a359-4ff2c577b32c" containerID="eeba465b1217eb955d9365e07d7f12c6f92682e2f4a6c26e814393a39ff78c52" exitCode=0 Nov 03 20:58:20.966076 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.965997 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" Nov 03 20:58:20.966076 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.966034 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" event={"ID":"038c8d67-7cd3-48bf-a359-4ff2c577b32c","Type":"ContainerDied","Data":"eeba465b1217eb955d9365e07d7f12c6f92682e2f4a6c26e814393a39ff78c52"} Nov 03 20:58:20.966076 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.966073 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx" event={"ID":"038c8d67-7cd3-48bf-a359-4ff2c577b32c","Type":"ContainerDied","Data":"ec9fc930b223318c293c2b3f57405e153b139a4c8e0df2b83dcbc08a817c24c8"} Nov 03 20:58:20.973735 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.973608 4493 scope.go:122] "RemoveContainer" containerID="0f03cd8d3f2fc52c722f700adc7a918ffdbde06257c42bb1f473f71fc5fbe371" Nov 03 20:58:20.989990 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.989972 4493 scope.go:122] "RemoveContainer" containerID="0f03cd8d3f2fc52c722f700adc7a918ffdbde06257c42bb1f473f71fc5fbe371" Nov 03 20:58:20.990232 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:58:20.990213 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f03cd8d3f2fc52c722f700adc7a918ffdbde06257c42bb1f473f71fc5fbe371\": container with ID starting with 0f03cd8d3f2fc52c722f700adc7a918ffdbde06257c42bb1f473f71fc5fbe371 not found: ID does not exist" containerID="0f03cd8d3f2fc52c722f700adc7a918ffdbde06257c42bb1f473f71fc5fbe371" Nov 03 20:58:20.990315 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.990239 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f03cd8d3f2fc52c722f700adc7a918ffdbde06257c42bb1f473f71fc5fbe371"} err="failed to get container status \"0f03cd8d3f2fc52c722f700adc7a918ffdbde06257c42bb1f473f71fc5fbe371\": rpc error: code = NotFound desc = could not find container \"0f03cd8d3f2fc52c722f700adc7a918ffdbde06257c42bb1f473f71fc5fbe371\": container with ID starting with 0f03cd8d3f2fc52c722f700adc7a918ffdbde06257c42bb1f473f71fc5fbe371 not found: ID does not exist" Nov 03 20:58:20.990315 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.990263 4493 scope.go:122] "RemoveContainer" containerID="f7006bd03df5896a554dfa898315f2fdbbdedb9a734b66264d360ccbe4488e1d" Nov 03 20:58:20.999628 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:20.999602 4493 scope.go:122] "RemoveContainer" containerID="e803eaca8188de5bd61537a843c8516a7a3c35cfd45188c4e8dc7b0c19f587a1" Nov 03 20:58:21.009591 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:21.009570 4493 scope.go:122] "RemoveContainer" containerID="f7006bd03df5896a554dfa898315f2fdbbdedb9a734b66264d360ccbe4488e1d" Nov 03 20:58:21.009873 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:58:21.009853 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7006bd03df5896a554dfa898315f2fdbbdedb9a734b66264d360ccbe4488e1d\": container with ID starting with f7006bd03df5896a554dfa898315f2fdbbdedb9a734b66264d360ccbe4488e1d not found: ID does not exist" containerID="f7006bd03df5896a554dfa898315f2fdbbdedb9a734b66264d360ccbe4488e1d" Nov 03 20:58:21.009927 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:21.009881 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7006bd03df5896a554dfa898315f2fdbbdedb9a734b66264d360ccbe4488e1d"} err="failed to get container status \"f7006bd03df5896a554dfa898315f2fdbbdedb9a734b66264d360ccbe4488e1d\": rpc error: code = NotFound desc = could not find container \"f7006bd03df5896a554dfa898315f2fdbbdedb9a734b66264d360ccbe4488e1d\": container with ID starting with f7006bd03df5896a554dfa898315f2fdbbdedb9a734b66264d360ccbe4488e1d not found: ID does not exist" Nov 03 20:58:21.009927 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:21.009902 4493 scope.go:122] "RemoveContainer" containerID="e803eaca8188de5bd61537a843c8516a7a3c35cfd45188c4e8dc7b0c19f587a1" Nov 03 20:58:21.010129 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:58:21.010114 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e803eaca8188de5bd61537a843c8516a7a3c35cfd45188c4e8dc7b0c19f587a1\": container with ID starting with e803eaca8188de5bd61537a843c8516a7a3c35cfd45188c4e8dc7b0c19f587a1 not found: ID does not exist" containerID="e803eaca8188de5bd61537a843c8516a7a3c35cfd45188c4e8dc7b0c19f587a1" Nov 03 20:58:21.010196 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:21.010133 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e803eaca8188de5bd61537a843c8516a7a3c35cfd45188c4e8dc7b0c19f587a1"} err="failed to get container status \"e803eaca8188de5bd61537a843c8516a7a3c35cfd45188c4e8dc7b0c19f587a1\": rpc error: code = NotFound desc = could not find container \"e803eaca8188de5bd61537a843c8516a7a3c35cfd45188c4e8dc7b0c19f587a1\": container with ID starting with e803eaca8188de5bd61537a843c8516a7a3c35cfd45188c4e8dc7b0c19f587a1 not found: ID does not exist" Nov 03 20:58:21.010196 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:21.010146 4493 scope.go:122] "RemoveContainer" containerID="eeba465b1217eb955d9365e07d7f12c6f92682e2f4a6c26e814393a39ff78c52" Nov 03 20:58:21.019597 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:21.019576 4493 scope.go:122] "RemoveContainer" containerID="eeba465b1217eb955d9365e07d7f12c6f92682e2f4a6c26e814393a39ff78c52" Nov 03 20:58:21.019863 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:58:21.019844 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeba465b1217eb955d9365e07d7f12c6f92682e2f4a6c26e814393a39ff78c52\": container with ID starting with eeba465b1217eb955d9365e07d7f12c6f92682e2f4a6c26e814393a39ff78c52 not found: ID does not exist" containerID="eeba465b1217eb955d9365e07d7f12c6f92682e2f4a6c26e814393a39ff78c52" Nov 03 20:58:21.019928 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:21.019868 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeba465b1217eb955d9365e07d7f12c6f92682e2f4a6c26e814393a39ff78c52"} err="failed to get container status \"eeba465b1217eb955d9365e07d7f12c6f92682e2f4a6c26e814393a39ff78c52\": rpc error: code = NotFound desc = could not find container \"eeba465b1217eb955d9365e07d7f12c6f92682e2f4a6c26e814393a39ff78c52\": container with ID starting with eeba465b1217eb955d9365e07d7f12c6f92682e2f4a6c26e814393a39ff78c52 not found: ID does not exist" Nov 03 20:58:21.115815 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:21.115745 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc"] Nov 03 20:58:21.162186 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:21.162157 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-7f6pc"] Nov 03 20:58:21.219924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:21.219841 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779"] Nov 03 20:58:21.224697 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:21.224671 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-5v779"] Nov 03 20:58:21.328955 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:21.328920 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx"] Nov 03 20:58:21.333021 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:21.332997 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hpzwx"] Nov 03 20:58:21.439768 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:21.439680 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm"] Nov 03 20:58:21.443156 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:21.443131 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-8mqjm"] Nov 03 20:58:21.745157 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:58:21.745077 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:58:22.562821 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:22.562786 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="038c8d67-7cd3-48bf-a359-4ff2c577b32c" path="/var/lib/kubelet/pods/038c8d67-7cd3-48bf-a359-4ff2c577b32c/volumes" Nov 03 20:58:22.563464 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:22.563449 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="156d066a-c5e2-469e-86b3-b92e319f71a0" path="/var/lib/kubelet/pods/156d066a-c5e2-469e-86b3-b92e319f71a0/volumes" Nov 03 20:58:22.563939 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:22.563925 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48c79419-fc27-4ffb-8222-63e30302cef4" path="/var/lib/kubelet/pods/48c79419-fc27-4ffb-8222-63e30302cef4/volumes" Nov 03 20:58:22.564387 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:22.564374 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3fb9508-37d2-47fa-bb02-ae7735c9f408" path="/var/lib/kubelet/pods/a3fb9508-37d2-47fa-bb02-ae7735c9f408/volumes" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.739437 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs"] Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.741740 4493 cpu_manager.go:414] "containerMap: removing container" podUID="156d066a-c5e2-469e-86b3-b92e319f71a0" containerName="istio-proxy" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.741762 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="156d066a-c5e2-469e-86b3-b92e319f71a0" containerName="istio-proxy" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.741791 4493 cpu_manager.go:414] "containerMap: removing container" podUID="a3fb9508-37d2-47fa-bb02-ae7735c9f408" containerName="istio-proxy" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.741799 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3fb9508-37d2-47fa-bb02-ae7735c9f408" containerName="istio-proxy" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.741809 4493 cpu_manager.go:414] "containerMap: removing container" podUID="a3fb9508-37d2-47fa-bb02-ae7735c9f408" containerName="istio-proxy" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.741818 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3fb9508-37d2-47fa-bb02-ae7735c9f408" containerName="istio-proxy" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.741947 4493 cpu_manager.go:414] "containerMap: removing container" podUID="fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb" containerName="istio-proxy" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.742018 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb" containerName="istio-proxy" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.742035 4493 cpu_manager.go:414] "containerMap: removing container" podUID="151eca7e-97d2-4220-a630-71e7df7d9f24" containerName="istio-proxy" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.742042 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151eca7e-97d2-4220-a630-71e7df7d9f24" containerName="istio-proxy" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.742054 4493 cpu_manager.go:414] "containerMap: removing container" podUID="d59bd1c6-7591-4470-b328-b0d60fb6f292" containerName="payload-pre-processing" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.742068 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59bd1c6-7591-4470-b328-b0d60fb6f292" containerName="payload-pre-processing" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.742103 4493 cpu_manager.go:414] "containerMap: removing container" podUID="b4e6937f-8195-42d2-a7e7-d14e98aba9cf" containerName="istio-proxy" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.742110 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e6937f-8195-42d2-a7e7-d14e98aba9cf" containerName="istio-proxy" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.742139 4493 cpu_manager.go:414] "containerMap: removing container" podUID="151eca7e-97d2-4220-a630-71e7df7d9f24" containerName="istio-proxy" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.742146 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151eca7e-97d2-4220-a630-71e7df7d9f24" containerName="istio-proxy" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.742170 4493 cpu_manager.go:414] "containerMap: removing container" podUID="f0ae7b36-9635-4ea7-badb-a5a97321c841" containerName="payload-processing" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.742177 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0ae7b36-9635-4ea7-badb-a5a97321c841" containerName="payload-processing" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.742198 4493 cpu_manager.go:414] "containerMap: removing container" podUID="f62e8863-2c8c-4658-8ecf-645709ec29e2" containerName="istio-proxy" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.742204 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f62e8863-2c8c-4658-8ecf-645709ec29e2" containerName="istio-proxy" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.742225 4493 cpu_manager.go:414] "containerMap: removing container" podUID="a47bd78c-2c95-4eb1-a2a3-b151a424ebdc" containerName="istio-proxy" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.742231 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47bd78c-2c95-4eb1-a2a3-b151a424ebdc" containerName="istio-proxy" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.742258 4493 cpu_manager.go:414] "containerMap: removing container" podUID="a47bd78c-2c95-4eb1-a2a3-b151a424ebdc" containerName="istio-proxy" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.742264 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47bd78c-2c95-4eb1-a2a3-b151a424ebdc" containerName="istio-proxy" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.742283 4493 cpu_manager.go:414] "containerMap: removing container" podUID="038c8d67-7cd3-48bf-a359-4ff2c577b32c" containerName="istio-proxy" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.742289 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="038c8d67-7cd3-48bf-a359-4ff2c577b32c" containerName="istio-proxy" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.742316 4493 cpu_manager.go:414] "containerMap: removing container" podUID="48c79419-fc27-4ffb-8222-63e30302cef4" containerName="istio-proxy" Nov 03 20:58:25.742547 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.742323 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48c79419-fc27-4ffb-8222-63e30302cef4" containerName="istio-proxy" Nov 03 20:58:25.744566 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.744538 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd923b0-6c5e-40d8-b5ba-d3c8e87d24eb" containerName="istio-proxy" Nov 03 20:58:25.744669 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.744578 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="151eca7e-97d2-4220-a630-71e7df7d9f24" containerName="istio-proxy" Nov 03 20:58:25.744669 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.744589 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59bd1c6-7591-4470-b328-b0d60fb6f292" containerName="payload-pre-processing" Nov 03 20:58:25.744669 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.744610 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e6937f-8195-42d2-a7e7-d14e98aba9cf" containerName="istio-proxy" Nov 03 20:58:25.744669 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.744628 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="151eca7e-97d2-4220-a630-71e7df7d9f24" containerName="istio-proxy" Nov 03 20:58:25.744669 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.744640 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0ae7b36-9635-4ea7-badb-a5a97321c841" containerName="payload-processing" Nov 03 20:58:25.744669 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.744655 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f62e8863-2c8c-4658-8ecf-645709ec29e2" containerName="istio-proxy" Nov 03 20:58:25.744669 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.744663 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47bd78c-2c95-4eb1-a2a3-b151a424ebdc" containerName="istio-proxy" Nov 03 20:58:25.744913 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.744673 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47bd78c-2c95-4eb1-a2a3-b151a424ebdc" containerName="istio-proxy" Nov 03 20:58:25.744913 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.744689 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="038c8d67-7cd3-48bf-a359-4ff2c577b32c" containerName="istio-proxy" Nov 03 20:58:25.744913 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.744709 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="48c79419-fc27-4ffb-8222-63e30302cef4" containerName="istio-proxy" Nov 03 20:58:25.744913 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.744732 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="156d066a-c5e2-469e-86b3-b92e319f71a0" containerName="istio-proxy" Nov 03 20:58:25.744913 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.744743 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3fb9508-37d2-47fa-bb02-ae7735c9f408" containerName="istio-proxy" Nov 03 20:58:25.744913 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.744757 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3fb9508-37d2-47fa-bb02-ae7735c9f408" containerName="istio-proxy" Nov 03 20:58:25.753494 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.753469 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs" Nov 03 20:58:25.798447 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.798415 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs"] Nov 03 20:58:25.838797 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.838761 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-b6d800\"" Nov 03 20:58:25.851876 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.851838 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-b6d800-dockercfg-mwtlw\"" Nov 03 20:58:25.890124 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.890094 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j"] Nov 03 20:58:25.891179 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.891152 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djjnd\" (UniqueName: \"kubernetes.io/projected/7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac-kube-api-access-djjnd\") pod \"payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs\" (UID: \"7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs" Nov 03 20:58:25.891391 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.891325 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs\" (UID: \"7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs" Nov 03 20:58:25.894325 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.894310 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j" Nov 03 20:58:25.944997 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.944946 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j"] Nov 03 20:58:26.001409 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.994882 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mjs8\" (UniqueName: \"kubernetes.io/projected/33d6f310-96b9-4029-8f11-179458b5b17c-kube-api-access-2mjs8\") pod \"payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j\" (UID: \"33d6f310-96b9-4029-8f11-179458b5b17c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j" Nov 03 20:58:26.001409 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.995048 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djjnd\" (UniqueName: \"kubernetes.io/projected/7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac-kube-api-access-djjnd\") pod \"payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs\" (UID: \"7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs" Nov 03 20:58:26.001409 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.995554 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33d6f310-96b9-4029-8f11-179458b5b17c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j\" (UID: \"33d6f310-96b9-4029-8f11-179458b5b17c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j" Nov 03 20:58:26.001409 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:25.995778 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs\" (UID: \"7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs" Nov 03 20:58:26.012847 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:26.012815 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs\" (UID: \"7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs" Nov 03 20:58:26.050349 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:26.049452 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djjnd\" (UniqueName: \"kubernetes.io/projected/7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac-kube-api-access-djjnd\") pod \"payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs\" (UID: \"7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs" Nov 03 20:58:26.069473 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:26.069446 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs" Nov 03 20:58:26.125146 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:26.125110 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mjs8\" (UniqueName: \"kubernetes.io/projected/33d6f310-96b9-4029-8f11-179458b5b17c-kube-api-access-2mjs8\") pod \"payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j\" (UID: \"33d6f310-96b9-4029-8f11-179458b5b17c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j" Nov 03 20:58:26.125279 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:26.125173 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33d6f310-96b9-4029-8f11-179458b5b17c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j\" (UID: \"33d6f310-96b9-4029-8f11-179458b5b17c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j" Nov 03 20:58:26.126006 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:26.125978 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33d6f310-96b9-4029-8f11-179458b5b17c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j\" (UID: \"33d6f310-96b9-4029-8f11-179458b5b17c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j" Nov 03 20:58:26.183506 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:26.183467 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mjs8\" (UniqueName: \"kubernetes.io/projected/33d6f310-96b9-4029-8f11-179458b5b17c-kube-api-access-2mjs8\") pod \"payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j\" (UID: \"33d6f310-96b9-4029-8f11-179458b5b17c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j" Nov 03 20:58:26.210678 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:26.210641 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j" Nov 03 20:58:26.457672 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:26.457615 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs"] Nov 03 20:58:26.603100 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:26.603061 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j"] Nov 03 20:58:26.611366 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:58:26.611067 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d6f310_96b9_4029_8f11_179458b5b17c.slice/crio-29b1b10a422f789f740478fb83ef4d12c9eb06f31f83e1690f9c3aca06ec886a WatchSource:0}: Error finding container 29b1b10a422f789f740478fb83ef4d12c9eb06f31f83e1690f9c3aca06ec886a: Status 404 returned error can't find the container with id 29b1b10a422f789f740478fb83ef4d12c9eb06f31f83e1690f9c3aca06ec886a Nov 03 20:58:27.023796 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:27.023697 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j" event={"ID":"33d6f310-96b9-4029-8f11-179458b5b17c","Type":"ContainerStarted","Data":"09693fe0550ab236a1310a6225b9e0493e01757bb48857dbd81665eddb7efa9d"} Nov 03 20:58:27.023796 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:27.023768 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j" Nov 03 20:58:27.024371 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:27.023809 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j" event={"ID":"33d6f310-96b9-4029-8f11-179458b5b17c","Type":"ContainerStarted","Data":"29b1b10a422f789f740478fb83ef4d12c9eb06f31f83e1690f9c3aca06ec886a"} Nov 03 20:58:27.025417 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:27.025387 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs" event={"ID":"7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac","Type":"ContainerStarted","Data":"2d1670eafa1fb278031879047fffb9716598bac0a2eeab898efed064fd47b810"} Nov 03 20:58:27.025554 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:27.025435 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs" event={"ID":"7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac","Type":"ContainerStarted","Data":"04667b727fc2f3373a58e89575a47b6d63d81f76a9b2d7c8c64dee2ab7ad6bd5"} Nov 03 20:58:27.025554 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:27.025459 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs" Nov 03 20:58:27.132937 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:27.132884 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j" podStartSLOduration=2.132869616 podStartE2EDuration="2.132869616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:27.13098043 +0000 UTC m=+1778.688970954" watchObservedRunningTime="2026-11-03 20:58:27.132869616 +0000 UTC m=+1778.690860116" Nov 03 20:58:27.251370 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:27.247996 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs" podStartSLOduration=2.247970313 podStartE2EDuration="2.247970313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:27.246894013 +0000 UTC m=+1778.804884531" watchObservedRunningTime="2026-11-03 20:58:27.247970313 +0000 UTC m=+1778.805960813" Nov 03 20:58:28.307603 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:28.307574 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4"] Nov 03 20:58:28.308089 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:28.307781 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" podUID="0d3000f1-172d-4b7b-9d45-fa04ef63145f" containerName="istio-proxy" containerID="cri-o://470fd3c29321dd66952480f1ca289ec06aa43e0b0d3bc6ea90f11048caadd103" gracePeriod=30 Nov 03 20:58:28.701566 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:28.701491 4493 patch_prober.go:29] interesting pod/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:58:28.701566 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:28.701538 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" podUID="0d3000f1-172d-4b7b-9d45-fa04ef63145f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:58:29.191754 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.191715 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5"] Nov 03 20:58:29.199370 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.199316 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.250222 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.250187 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-1e862b-openshift-default-dockercfg-zzvts\"" Nov 03 20:58:29.302578 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.302540 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istiod-ca-cert\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.302740 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.302679 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-data\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.302740 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.302732 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-workload-certs\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.302856 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.302775 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-credential-socket\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.302944 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.302926 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-token\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.303033 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.302978 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-podinfo\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.303033 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.303011 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxpjk\" (UniqueName: \"kubernetes.io/projected/c73e8310-a7fd-439b-8f6a-95e99d528e5d-kube-api-access-nxpjk\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.303211 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.303188 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-workload-socket\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.303327 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.303309 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-envoy\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.303891 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.303870 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5"] Nov 03 20:58:29.404433 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.404400 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-envoy\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.404908 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.404471 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istiod-ca-cert\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.404908 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.404508 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-data\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.404908 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.404530 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-workload-certs\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.404908 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.404573 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-credential-socket\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.404908 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.404636 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-token\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.404908 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.404678 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-podinfo\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.404908 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.404708 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxpjk\" (UniqueName: \"kubernetes.io/projected/c73e8310-a7fd-439b-8f6a-95e99d528e5d-kube-api-access-nxpjk\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.404908 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.404850 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-workload-socket\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.405319 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.404961 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-workload-certs\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.405319 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.405023 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-credential-socket\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.405319 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.405217 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-workload-socket\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.405319 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.405244 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-data\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.405488 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.405401 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istiod-ca-cert\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.407685 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.407658 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-envoy\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.409936 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.407920 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-podinfo\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.458761 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.458697 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-token\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.459077 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.459057 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxpjk\" (UniqueName: \"kubernetes.io/projected/c73e8310-a7fd-439b-8f6a-95e99d528e5d-kube-api-access-nxpjk\") pod \"e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.517015 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.516986 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:29.953184 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.953150 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5"] Nov 03 20:58:29.957160 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:58:29.957126 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73e8310_a7fd_439b_8f6a_95e99d528e5d.slice/crio-3c13484ca5f260a742d8892bded4bcab8fd359f0630601e561e4fd3a6523c3f8 WatchSource:0}: Error finding container 3c13484ca5f260a742d8892bded4bcab8fd359f0630601e561e4fd3a6523c3f8: Status 404 returned error can't find the container with id 3c13484ca5f260a742d8892bded4bcab8fd359f0630601e561e4fd3a6523c3f8 Nov 03 20:58:29.960191 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.960157 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:58:29.960274 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.960226 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:58:29.960274 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:29.960255 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:58:30.049578 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.049543 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" event={"ID":"c73e8310-a7fd-439b-8f6a-95e99d528e5d","Type":"ContainerStarted","Data":"9164054bb75d12a8e0d3943ad1a0b1c91cc7ac038ac21cfd3fea388053a265b0"} Nov 03 20:58:30.049750 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.049598 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" event={"ID":"c73e8310-a7fd-439b-8f6a-95e99d528e5d","Type":"ContainerStarted","Data":"3c13484ca5f260a742d8892bded4bcab8fd359f0630601e561e4fd3a6523c3f8"} Nov 03 20:58:30.168153 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.168107 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" podStartSLOduration=1.168092677 podStartE2EDuration="1.168092677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:30.163501517 +0000 UTC m=+1781.721492036" watchObservedRunningTime="2026-11-03 20:58:30.168092677 +0000 UTC m=+1781.726083196" Nov 03 20:58:30.517542 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.517507 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:30.703736 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.703704 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6"] Nov 03 20:58:30.707917 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.707900 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.757159 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.757103 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-1ec1b6-openshift-default-dockercfg-5trqn\"" Nov 03 20:58:30.787780 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.787690 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-podinfo\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.787953 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.787786 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-workload-socket\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.787953 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.787923 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-workload-certs\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.788066 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.788046 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-envoy\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.788155 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.788136 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-token\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.788214 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.788183 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx2dz\" (UniqueName: \"kubernetes.io/projected/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-kube-api-access-nx2dz\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.788268 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.788237 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-data\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.788323 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.788286 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istiod-ca-cert\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.788517 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.788500 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-credential-socket\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.808222 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.808188 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6"] Nov 03 20:58:30.896816 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.896770 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-envoy\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.897029 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.896929 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-token\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.897029 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.896974 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx2dz\" (UniqueName: \"kubernetes.io/projected/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-kube-api-access-nx2dz\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.897029 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.897013 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-data\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.897201 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.897042 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istiod-ca-cert\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.897255 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.897209 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-credential-socket\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.897306 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.897290 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-podinfo\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.897464 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.897442 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-workload-socket\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.897624 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.897574 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-workload-certs\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.898070 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.898043 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-workload-certs\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.898584 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.898557 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-credential-socket\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.898791 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.898764 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istiod-ca-cert\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.898862 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.898837 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-data\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.899110 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.899082 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-workload-socket\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.899691 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.899669 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-envoy\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.901031 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.901011 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-podinfo\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.953224 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.953181 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx2dz\" (UniqueName: \"kubernetes.io/projected/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-kube-api-access-nx2dz\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:30.953961 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:30.953937 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-token\") pod \"e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:31.025057 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:31.025019 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:31.467201 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:31.467172 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6"] Nov 03 20:58:31.471022 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:58:31.470992 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode51960f4_5af5_4d6b_ae00_29b0e54c5ba3.slice/crio-3538c56d951d3b25b0b4c4c1fe4a67fc1f861d983cee4653d8fe6170561bf113 WatchSource:0}: Error finding container 3538c56d951d3b25b0b4c4c1fe4a67fc1f861d983cee4653d8fe6170561bf113: Status 404 returned error can't find the container with id 3538c56d951d3b25b0b4c4c1fe4a67fc1f861d983cee4653d8fe6170561bf113 Nov 03 20:58:31.473712 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:31.473679 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:58:31.473812 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:31.473749 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:58:31.473812 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:31.473777 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:58:31.524513 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:31.524490 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:32.068916 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:32.068872 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" event={"ID":"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3","Type":"ContainerStarted","Data":"c566abe6eaaae1012a09bfbac4215eaed8a252ee2323151f7334bcb5c595faaf"} Nov 03 20:58:32.069124 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:32.068977 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" event={"ID":"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3","Type":"ContainerStarted","Data":"3538c56d951d3b25b0b4c4c1fe4a67fc1f861d983cee4653d8fe6170561bf113"} Nov 03 20:58:32.069192 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:32.069144 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:32.070219 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:32.070197 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 20:58:32.183705 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:32.183654 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" podStartSLOduration=2.183639145 podStartE2EDuration="2.183639145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:32.178868988 +0000 UTC m=+1783.736859515" watchObservedRunningTime="2026-11-03 20:58:32.183639145 +0000 UTC m=+1783.741629644" Nov 03 20:58:33.026236 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.026205 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:33.032926 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.032901 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:33.076993 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.076960 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:33.077812 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.077792 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 20:58:33.586935 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.586879 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:58:33.640557 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.640519 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-envoy\") pod \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " Nov 03 20:58:33.640729 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.640574 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-podinfo\") pod \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " Nov 03 20:58:33.640729 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.640616 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d3000f1-172d-4b7b-9d45-fa04ef63145f-kube-api-access-94sq6\" (UniqueName: \"kubernetes.io/projected/0d3000f1-172d-4b7b-9d45-fa04ef63145f-kube-api-access-94sq6\") pod \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " Nov 03 20:58:33.640729 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.640706 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-workload-socket\") pod \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " Nov 03 20:58:33.640899 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.640850 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-workload-certs\") pod \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " Nov 03 20:58:33.640899 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.640887 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-credential-socket\") pod \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " Nov 03 20:58:33.640999 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.640977 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-data\") pod \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " Nov 03 20:58:33.641060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.641012 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istiod-ca-cert\") pod \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " Nov 03 20:58:33.641060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.641048 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-token\" (UniqueName: \"kubernetes.io/projected/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-token\") pod \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\" (UID: \"0d3000f1-172d-4b7b-9d45-fa04ef63145f\") " Nov 03 20:58:33.641180 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.641155 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-workload-certs" pod "0d3000f1-172d-4b7b-9d45-fa04ef63145f" (UID: "0d3000f1-172d-4b7b-9d45-fa04ef63145f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:33.641438 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.641285 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-data" pod "0d3000f1-172d-4b7b-9d45-fa04ef63145f" (UID: "0d3000f1-172d-4b7b-9d45-fa04ef63145f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:33.641438 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.641405 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-credential-socket" pod "0d3000f1-172d-4b7b-9d45-fa04ef63145f" (UID: "0d3000f1-172d-4b7b-9d45-fa04ef63145f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:33.641619 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.641485 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-workload-socket" pod "0d3000f1-172d-4b7b-9d45-fa04ef63145f" (UID: "0d3000f1-172d-4b7b-9d45-fa04ef63145f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:33.641619 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.641508 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istiod-ca-cert" pod "0d3000f1-172d-4b7b-9d45-fa04ef63145f" (UID: "0d3000f1-172d-4b7b-9d45-fa04ef63145f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:33.642024 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.641996 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:33.642126 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.642030 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:33.642126 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.642045 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:33.642126 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.642060 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:33.642126 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.642073 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:33.643081 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.643041 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-envoy" pod "0d3000f1-172d-4b7b-9d45-fa04ef63145f" (UID: "0d3000f1-172d-4b7b-9d45-fa04ef63145f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:33.643574 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.643547 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-token" pod "0d3000f1-172d-4b7b-9d45-fa04ef63145f" (UID: "0d3000f1-172d-4b7b-9d45-fa04ef63145f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:33.643935 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.643910 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-podinfo" pod "0d3000f1-172d-4b7b-9d45-fa04ef63145f" (UID: "0d3000f1-172d-4b7b-9d45-fa04ef63145f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:33.644278 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.644254 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3000f1-172d-4b7b-9d45-fa04ef63145f-kube-api-access-94sq6" pod "0d3000f1-172d-4b7b-9d45-fa04ef63145f" (UID: "0d3000f1-172d-4b7b-9d45-fa04ef63145f"). InnerVolumeSpecName "kube-api-access-94sq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:33.743121 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.743076 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:33.743121 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.743110 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:33.743121 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.743120 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94sq6\" (UniqueName: \"kubernetes.io/projected/0d3000f1-172d-4b7b-9d45-fa04ef63145f-kube-api-access-94sq6\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:33.743121 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:33.743130 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d3000f1-172d-4b7b-9d45-fa04ef63145f-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:34.085660 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:34.085626 4493 generic.go:356] "Generic (PLEG): container finished" podID="0d3000f1-172d-4b7b-9d45-fa04ef63145f" containerID="470fd3c29321dd66952480f1ca289ec06aa43e0b0d3bc6ea90f11048caadd103" exitCode=0 Nov 03 20:58:34.086123 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:34.085677 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" Nov 03 20:58:34.086123 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:34.085707 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" event={"ID":"0d3000f1-172d-4b7b-9d45-fa04ef63145f","Type":"ContainerDied","Data":"470fd3c29321dd66952480f1ca289ec06aa43e0b0d3bc6ea90f11048caadd103"} Nov 03 20:58:34.086123 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:34.085770 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4" event={"ID":"0d3000f1-172d-4b7b-9d45-fa04ef63145f","Type":"ContainerDied","Data":"abf671b0bc7c7aa34a6b013de37db3fc2f4d1b670973d1650b1a5b8b5293ad2b"} Nov 03 20:58:34.086123 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:34.085796 4493 scope.go:122] "RemoveContainer" containerID="470fd3c29321dd66952480f1ca289ec06aa43e0b0d3bc6ea90f11048caadd103" Nov 03 20:58:34.096827 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:34.096809 4493 scope.go:122] "RemoveContainer" containerID="ec76955c5127e08a3ce1b7f2d997987b8d7360848ce06ce1a81ce2fcb71cc88a" Nov 03 20:58:34.106223 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:34.106206 4493 scope.go:122] "RemoveContainer" containerID="470fd3c29321dd66952480f1ca289ec06aa43e0b0d3bc6ea90f11048caadd103" Nov 03 20:58:34.106584 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:58:34.106566 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"470fd3c29321dd66952480f1ca289ec06aa43e0b0d3bc6ea90f11048caadd103\": container with ID starting with 470fd3c29321dd66952480f1ca289ec06aa43e0b0d3bc6ea90f11048caadd103 not found: ID does not exist" containerID="470fd3c29321dd66952480f1ca289ec06aa43e0b0d3bc6ea90f11048caadd103" Nov 03 20:58:34.106637 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:34.106594 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"470fd3c29321dd66952480f1ca289ec06aa43e0b0d3bc6ea90f11048caadd103"} err="failed to get container status \"470fd3c29321dd66952480f1ca289ec06aa43e0b0d3bc6ea90f11048caadd103\": rpc error: code = NotFound desc = could not find container \"470fd3c29321dd66952480f1ca289ec06aa43e0b0d3bc6ea90f11048caadd103\": container with ID starting with 470fd3c29321dd66952480f1ca289ec06aa43e0b0d3bc6ea90f11048caadd103 not found: ID does not exist" Nov 03 20:58:34.106637 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:34.106610 4493 scope.go:122] "RemoveContainer" containerID="ec76955c5127e08a3ce1b7f2d997987b8d7360848ce06ce1a81ce2fcb71cc88a" Nov 03 20:58:34.106888 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:58:34.106867 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec76955c5127e08a3ce1b7f2d997987b8d7360848ce06ce1a81ce2fcb71cc88a\": container with ID starting with ec76955c5127e08a3ce1b7f2d997987b8d7360848ce06ce1a81ce2fcb71cc88a not found: ID does not exist" containerID="ec76955c5127e08a3ce1b7f2d997987b8d7360848ce06ce1a81ce2fcb71cc88a" Nov 03 20:58:34.106937 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:34.106898 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec76955c5127e08a3ce1b7f2d997987b8d7360848ce06ce1a81ce2fcb71cc88a"} err="failed to get container status \"ec76955c5127e08a3ce1b7f2d997987b8d7360848ce06ce1a81ce2fcb71cc88a\": rpc error: code = NotFound desc = could not find container \"ec76955c5127e08a3ce1b7f2d997987b8d7360848ce06ce1a81ce2fcb71cc88a\": container with ID starting with ec76955c5127e08a3ce1b7f2d997987b8d7360848ce06ce1a81ce2fcb71cc88a not found: ID does not exist" Nov 03 20:58:34.243902 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:34.243872 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4"] Nov 03 20:58:34.289721 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:34.289691 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-cc15b1d7-openshift-default-6d8c548d45-tkpg4"] Nov 03 20:58:34.566731 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:34.566699 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d3000f1-172d-4b7b-9d45-fa04ef63145f" path="/var/lib/kubelet/pods/0d3000f1-172d-4b7b-9d45-fa04ef63145f/volumes" Nov 03 20:58:37.947003 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:37.946966 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb"] Nov 03 20:58:37.947865 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:37.947841 4493 cpu_manager.go:414] "containerMap: removing container" podUID="0d3000f1-172d-4b7b-9d45-fa04ef63145f" containerName="istio-proxy" Nov 03 20:58:37.947978 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:37.947866 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d3000f1-172d-4b7b-9d45-fa04ef63145f" containerName="istio-proxy" Nov 03 20:58:37.948121 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:37.948106 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d3000f1-172d-4b7b-9d45-fa04ef63145f" containerName="istio-proxy" Nov 03 20:58:37.948185 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:37.948128 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d3000f1-172d-4b7b-9d45-fa04ef63145f" containerName="istio-proxy" Nov 03 20:58:37.953484 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:37.953460 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb" Nov 03 20:58:38.007783 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:38.007748 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb"] Nov 03 20:58:38.069848 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:38.069815 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-1e862b-dockercfg-6xqkv\"" Nov 03 20:58:38.070512 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:38.070483 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-1e862b\"" Nov 03 20:58:38.072792 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:38.071850 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea30227a-a99c-4d5f-ba2b-3e6230c29c2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb\" (UID: \"ea30227a-a99c-4d5f-ba2b-3e6230c29c2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb" Nov 03 20:58:38.072792 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:38.071921 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4v2v\" (UniqueName: \"kubernetes.io/projected/ea30227a-a99c-4d5f-ba2b-3e6230c29c2e-kube-api-access-c4v2v\") pod \"payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb\" (UID: \"ea30227a-a99c-4d5f-ba2b-3e6230c29c2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb" Nov 03 20:58:38.088210 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:38.085984 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs" Nov 03 20:58:38.108199 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:38.105226 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g"] Nov 03 20:58:38.108199 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:38.106255 4493 cpu_manager.go:414] "containerMap: removing container" podUID="0d3000f1-172d-4b7b-9d45-fa04ef63145f" containerName="istio-proxy" Nov 03 20:58:38.108199 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:38.106278 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d3000f1-172d-4b7b-9d45-fa04ef63145f" containerName="istio-proxy" Nov 03 20:58:38.116465 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:38.116426 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j" Nov 03 20:58:38.116623 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:38.116548 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g" Nov 03 20:58:38.160142 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:38.160113 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g"] Nov 03 20:58:38.173531 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:38.173493 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87af162e-e529-43f8-9673-0af5fcdc9d7a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g\" (UID: \"87af162e-e529-43f8-9673-0af5fcdc9d7a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g" Nov 03 20:58:38.173682 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:38.173660 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea30227a-a99c-4d5f-ba2b-3e6230c29c2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb\" (UID: \"ea30227a-a99c-4d5f-ba2b-3e6230c29c2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb" Nov 03 20:58:38.173745 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:38.173729 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4v2v\" (UniqueName: \"kubernetes.io/projected/ea30227a-a99c-4d5f-ba2b-3e6230c29c2e-kube-api-access-c4v2v\") pod \"payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb\" (UID: \"ea30227a-a99c-4d5f-ba2b-3e6230c29c2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb" Nov 03 20:58:38.174015 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:38.173994 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k46ww\" (UniqueName: \"kubernetes.io/projected/87af162e-e529-43f8-9673-0af5fcdc9d7a-kube-api-access-k46ww\") pod \"payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g\" (UID: \"87af162e-e529-43f8-9673-0af5fcdc9d7a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g" Nov 03 20:58:38.175745 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:38.175718 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea30227a-a99c-4d5f-ba2b-3e6230c29c2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb\" (UID: \"ea30227a-a99c-4d5f-ba2b-3e6230c29c2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb" Nov 03 20:58:38.229452 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:38.229378 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4v2v\" (UniqueName: \"kubernetes.io/projected/ea30227a-a99c-4d5f-ba2b-3e6230c29c2e-kube-api-access-c4v2v\") pod \"payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb\" (UID: \"ea30227a-a99c-4d5f-ba2b-3e6230c29c2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb" Nov 03 20:58:38.268506 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:38.268471 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb" Nov 03 20:58:38.275846 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:38.275815 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k46ww\" (UniqueName: \"kubernetes.io/projected/87af162e-e529-43f8-9673-0af5fcdc9d7a-kube-api-access-k46ww\") pod \"payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g\" (UID: \"87af162e-e529-43f8-9673-0af5fcdc9d7a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g" Nov 03 20:58:38.275999 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:38.275899 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87af162e-e529-43f8-9673-0af5fcdc9d7a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g\" (UID: \"87af162e-e529-43f8-9673-0af5fcdc9d7a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g" Nov 03 20:58:38.276983 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:38.276957 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87af162e-e529-43f8-9673-0af5fcdc9d7a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g\" (UID: \"87af162e-e529-43f8-9673-0af5fcdc9d7a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g" Nov 03 20:58:38.336029 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:38.334707 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k46ww\" (UniqueName: \"kubernetes.io/projected/87af162e-e529-43f8-9673-0af5fcdc9d7a-kube-api-access-k46ww\") pod \"payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g\" (UID: \"87af162e-e529-43f8-9673-0af5fcdc9d7a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g" Nov 03 20:58:38.456635 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:38.456606 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g" Nov 03 20:58:38.672775 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:38.672695 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb"] Nov 03 20:58:38.675559 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:58:38.675522 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea30227a_a99c_4d5f_ba2b_3e6230c29c2e.slice/crio-8a83681fae2cb55320e430ad2fe71deb9a265e1673c6a6f06dfb764a621e5953 WatchSource:0}: Error finding container 8a83681fae2cb55320e430ad2fe71deb9a265e1673c6a6f06dfb764a621e5953: Status 404 returned error can't find the container with id 8a83681fae2cb55320e430ad2fe71deb9a265e1673c6a6f06dfb764a621e5953 Nov 03 20:58:38.885489 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:38.885454 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g"] Nov 03 20:58:38.891714 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:58:38.891690 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87af162e_e529_43f8_9673_0af5fcdc9d7a.slice/crio-43325ef54fb37269f62988842507e2c4440df3f2d11041d55685f8ab608fd542 WatchSource:0}: Error finding container 43325ef54fb37269f62988842507e2c4440df3f2d11041d55685f8ab608fd542: Status 404 returned error can't find the container with id 43325ef54fb37269f62988842507e2c4440df3f2d11041d55685f8ab608fd542 Nov 03 20:58:39.132814 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:39.132774 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g" event={"ID":"87af162e-e529-43f8-9673-0af5fcdc9d7a","Type":"ContainerStarted","Data":"b172430e99560b6ea9cccefc15795ef9c68f249abf21f90cd04959d8dff1d2d3"} Nov 03 20:58:39.133277 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:39.132835 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g" event={"ID":"87af162e-e529-43f8-9673-0af5fcdc9d7a","Type":"ContainerStarted","Data":"43325ef54fb37269f62988842507e2c4440df3f2d11041d55685f8ab608fd542"} Nov 03 20:58:39.133277 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:39.132857 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g" Nov 03 20:58:39.134695 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:39.134667 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb" event={"ID":"ea30227a-a99c-4d5f-ba2b-3e6230c29c2e","Type":"ContainerStarted","Data":"f54b102006160ef7a2218573ca363fb5322407ae62db4b3b52a22207aacee706"} Nov 03 20:58:39.134822 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:39.134719 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb" event={"ID":"ea30227a-a99c-4d5f-ba2b-3e6230c29c2e","Type":"ContainerStarted","Data":"8a83681fae2cb55320e430ad2fe71deb9a265e1673c6a6f06dfb764a621e5953"} Nov 03 20:58:39.134822 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:39.134747 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb" Nov 03 20:58:39.244532 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:39.244477 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g" podStartSLOduration=1.24445925 podStartE2EDuration="1.24445925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:39.241162227 +0000 UTC m=+1790.799152753" watchObservedRunningTime="2026-11-03 20:58:39.24445925 +0000 UTC m=+1790.802449752" Nov 03 20:58:39.357731 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:39.352314 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb" podStartSLOduration=2.352294465 podStartE2EDuration="2.352294465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:39.347108045 +0000 UTC m=+1790.905098565" watchObservedRunningTime="2026-11-03 20:58:39.352294465 +0000 UTC m=+1790.910284964" Nov 03 20:58:42.791234 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:42.791199 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt"] Nov 03 20:58:42.795605 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:42.795585 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt" Nov 03 20:58:42.840607 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:42.840572 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq"] Nov 03 20:58:42.840835 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:42.840810 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" podUID="353543b3-c2cc-4563-a2a0-ddcf5b03db1b" containerName="istio-proxy" containerID="cri-o://267268c7a56d422d88a164baa20584df3662c33b63e54ac87e30ba71e43b08f4" gracePeriod=30 Nov 03 20:58:42.872030 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:42.871992 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d12d909a-e17b-470b-9ba6-164d1dbffba1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt\" (UID: \"d12d909a-e17b-470b-9ba6-164d1dbffba1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt" Nov 03 20:58:42.872470 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:42.872422 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7krj5\" (UniqueName: \"kubernetes.io/projected/d12d909a-e17b-470b-9ba6-164d1dbffba1-kube-api-access-7krj5\") pod \"payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt\" (UID: \"d12d909a-e17b-470b-9ba6-164d1dbffba1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt" Nov 03 20:58:42.888443 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:42.888412 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc"] Nov 03 20:58:42.888710 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:42.888662 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" podUID="adf4f7e5-e60d-484e-823f-1a40134f48fc" containerName="istio-proxy" containerID="cri-o://6582c21ef2abd9d2ddab5c8f0f227b8a2355fbcb1cdea4d67c905f91f33452ea" gracePeriod=30 Nov 03 20:58:42.892437 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:42.892412 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g"] Nov 03 20:58:42.892783 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:42.892750 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" podUID="a092ea83-8438-4626-a3a6-0e076627ab11" containerName="istio-proxy" containerID="cri-o://9f89a58d3ee4de632bbd5db4bd9c9860bade2b236b687e01d2baa7636084ba27" gracePeriod=30 Nov 03 20:58:42.936361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:42.932045 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-1ec1b6-dockercfg-p2tp4\"" Nov 03 20:58:42.936361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:42.932152 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-1ec1b6\"" Nov 03 20:58:42.942668 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:42.942556 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b"] Nov 03 20:58:42.942932 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:42.942849 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" podUID="57aeddbf-a41f-4cc0-8f76-d589898c2144" containerName="istio-proxy" containerID="cri-o://9571d5e8003ba1be713011a8d27c1a7b39ddd944bbc126a134c01dff54c10474" gracePeriod=30 Nov 03 20:58:42.944485 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:42.944456 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt"] Nov 03 20:58:42.959370 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:42.959304 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj"] Nov 03 20:58:42.968543 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:42.968498 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj" Nov 03 20:58:42.991358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:42.974646 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d12d909a-e17b-470b-9ba6-164d1dbffba1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt\" (UID: \"d12d909a-e17b-470b-9ba6-164d1dbffba1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt" Nov 03 20:58:42.991358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:42.974938 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7krj5\" (UniqueName: \"kubernetes.io/projected/d12d909a-e17b-470b-9ba6-164d1dbffba1-kube-api-access-7krj5\") pod \"payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt\" (UID: \"d12d909a-e17b-470b-9ba6-164d1dbffba1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt" Nov 03 20:58:42.991358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:42.976250 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d12d909a-e17b-470b-9ba6-164d1dbffba1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt\" (UID: \"d12d909a-e17b-470b-9ba6-164d1dbffba1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt" Nov 03 20:58:43.053359 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:43.049878 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj"] Nov 03 20:58:43.085927 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:43.083387 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj\" (UID: \"f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj" Nov 03 20:58:43.086374 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:43.086323 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqrw2\" (UniqueName: \"kubernetes.io/projected/f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898-kube-api-access-rqrw2\") pod \"payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj\" (UID: \"f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj" Nov 03 20:58:43.148060 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:43.136983 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7krj5\" (UniqueName: \"kubernetes.io/projected/d12d909a-e17b-470b-9ba6-164d1dbffba1-kube-api-access-7krj5\") pod \"payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt\" (UID: \"d12d909a-e17b-470b-9ba6-164d1dbffba1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt" Nov 03 20:58:43.188548 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:43.188507 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqrw2\" (UniqueName: \"kubernetes.io/projected/f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898-kube-api-access-rqrw2\") pod \"payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj\" (UID: \"f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj" Nov 03 20:58:43.188833 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:43.188816 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj\" (UID: \"f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj" Nov 03 20:58:43.189563 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:43.189542 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj\" (UID: \"f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj" Nov 03 20:58:43.246473 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:43.246438 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqrw2\" (UniqueName: \"kubernetes.io/projected/f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898-kube-api-access-rqrw2\") pod \"payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj\" (UID: \"f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj" Nov 03 20:58:43.366750 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:43.366669 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj" Nov 03 20:58:43.411313 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:43.411276 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt" Nov 03 20:58:43.759387 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:43.759189 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj"] Nov 03 20:58:43.762738 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:58:43.762713 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9bf9f02_cd0d_4f3a_8aaa_d83cbd778898.slice/crio-ab764eb0257655056f357543cfa231c810fa8505447b6fcb823ae353c61b12f7 WatchSource:0}: Error finding container ab764eb0257655056f357543cfa231c810fa8505447b6fcb823ae353c61b12f7: Status 404 returned error can't find the container with id ab764eb0257655056f357543cfa231c810fa8505447b6fcb823ae353c61b12f7 Nov 03 20:58:43.825427 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:43.825387 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt"] Nov 03 20:58:43.829434 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:58:43.829414 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12d909a_e17b_470b_9ba6_164d1dbffba1.slice/crio-8118fa4a1ef21223bc9e89f2a08f45393fc6675806d9168bea4916215690aac3 WatchSource:0}: Error finding container 8118fa4a1ef21223bc9e89f2a08f45393fc6675806d9168bea4916215690aac3: Status 404 returned error can't find the container with id 8118fa4a1ef21223bc9e89f2a08f45393fc6675806d9168bea4916215690aac3 Nov 03 20:58:44.172561 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:44.172531 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt" event={"ID":"d12d909a-e17b-470b-9ba6-164d1dbffba1","Type":"ContainerStarted","Data":"d6e853a62dde0a69ab54e59c5ee0201026b92be49d7db3e85bf172577e797729"} Nov 03 20:58:44.172735 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:44.172593 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt" event={"ID":"d12d909a-e17b-470b-9ba6-164d1dbffba1","Type":"ContainerStarted","Data":"8118fa4a1ef21223bc9e89f2a08f45393fc6675806d9168bea4916215690aac3"} Nov 03 20:58:44.172735 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:44.172621 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt" Nov 03 20:58:44.174446 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:44.174421 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj" event={"ID":"f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898","Type":"ContainerStarted","Data":"7b2dd6ab4adba0820274b3cd2a930aa6f3e496d48573172d5c93b2736bec85dd"} Nov 03 20:58:44.174588 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:44.174460 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj" event={"ID":"f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898","Type":"ContainerStarted","Data":"ab764eb0257655056f357543cfa231c810fa8505447b6fcb823ae353c61b12f7"} Nov 03 20:58:44.174588 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:44.174549 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj" Nov 03 20:58:44.282148 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:44.282082 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt" podStartSLOduration=2.282069435 podStartE2EDuration="2.282069435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:44.278705829 +0000 UTC m=+1795.836696352" watchObservedRunningTime="2026-11-03 20:58:44.282069435 +0000 UTC m=+1795.840059933" Nov 03 20:58:44.391115 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:44.391050 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj" podStartSLOduration=2.391030927 podStartE2EDuration="2.391030927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:44.385474027 +0000 UTC m=+1795.943464549" watchObservedRunningTime="2026-11-03 20:58:44.391030927 +0000 UTC m=+1795.949021427" Nov 03 20:58:47.893947 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:47.893902 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:58:47.894285 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:47.893964 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" podUID="57aeddbf-a41f-4cc0-8f76-d589898c2144" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:58:48.195226 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.195201 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:58:48.207504 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.207474 4493 generic.go:356] "Generic (PLEG): container finished" podID="353543b3-c2cc-4563-a2a0-ddcf5b03db1b" containerID="267268c7a56d422d88a164baa20584df3662c33b63e54ac87e30ba71e43b08f4" exitCode=0 Nov 03 20:58:48.207793 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.207599 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" Nov 03 20:58:48.207793 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.207759 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" event={"ID":"353543b3-c2cc-4563-a2a0-ddcf5b03db1b","Type":"ContainerDied","Data":"267268c7a56d422d88a164baa20584df3662c33b63e54ac87e30ba71e43b08f4"} Nov 03 20:58:48.208507 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.208423 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq" event={"ID":"353543b3-c2cc-4563-a2a0-ddcf5b03db1b","Type":"ContainerDied","Data":"ba174acdf2f4730b2ca0835ac86301d6710e757a9c08628f8b34c3f93dc77410"} Nov 03 20:58:48.208894 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.208516 4493 scope.go:122] "RemoveContainer" containerID="267268c7a56d422d88a164baa20584df3662c33b63e54ac87e30ba71e43b08f4" Nov 03 20:58:48.211707 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.211681 4493 generic.go:356] "Generic (PLEG): container finished" podID="a092ea83-8438-4626-a3a6-0e076627ab11" containerID="9f89a58d3ee4de632bbd5db4bd9c9860bade2b236b687e01d2baa7636084ba27" exitCode=0 Nov 03 20:58:48.211818 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.211802 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" event={"ID":"a092ea83-8438-4626-a3a6-0e076627ab11","Type":"ContainerDied","Data":"9f89a58d3ee4de632bbd5db4bd9c9860bade2b236b687e01d2baa7636084ba27"} Nov 03 20:58:48.214080 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.214053 4493 generic.go:356] "Generic (PLEG): container finished" podID="adf4f7e5-e60d-484e-823f-1a40134f48fc" containerID="6582c21ef2abd9d2ddab5c8f0f227b8a2355fbcb1cdea4d67c905f91f33452ea" exitCode=0 Nov 03 20:58:48.214198 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.214150 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" event={"ID":"adf4f7e5-e60d-484e-823f-1a40134f48fc","Type":"ContainerDied","Data":"6582c21ef2abd9d2ddab5c8f0f227b8a2355fbcb1cdea4d67c905f91f33452ea"} Nov 03 20:58:48.216244 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.216128 4493 generic.go:356] "Generic (PLEG): container finished" podID="57aeddbf-a41f-4cc0-8f76-d589898c2144" containerID="9571d5e8003ba1be713011a8d27c1a7b39ddd944bbc126a134c01dff54c10474" exitCode=0 Nov 03 20:58:48.216244 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.216220 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" event={"ID":"57aeddbf-a41f-4cc0-8f76-d589898c2144","Type":"ContainerDied","Data":"9571d5e8003ba1be713011a8d27c1a7b39ddd944bbc126a134c01dff54c10474"} Nov 03 20:58:48.225050 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.225030 4493 scope.go:122] "RemoveContainer" containerID="267268c7a56d422d88a164baa20584df3662c33b63e54ac87e30ba71e43b08f4" Nov 03 20:58:48.225368 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:58:48.225316 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"267268c7a56d422d88a164baa20584df3662c33b63e54ac87e30ba71e43b08f4\": container with ID starting with 267268c7a56d422d88a164baa20584df3662c33b63e54ac87e30ba71e43b08f4 not found: ID does not exist" containerID="267268c7a56d422d88a164baa20584df3662c33b63e54ac87e30ba71e43b08f4" Nov 03 20:58:48.225445 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.225378 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"267268c7a56d422d88a164baa20584df3662c33b63e54ac87e30ba71e43b08f4"} err="failed to get container status \"267268c7a56d422d88a164baa20584df3662c33b63e54ac87e30ba71e43b08f4\": rpc error: code = NotFound desc = could not find container \"267268c7a56d422d88a164baa20584df3662c33b63e54ac87e30ba71e43b08f4\": container with ID starting with 267268c7a56d422d88a164baa20584df3662c33b63e54ac87e30ba71e43b08f4 not found: ID does not exist" Nov 03 20:58:48.225445 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.225401 4493 scope.go:122] "RemoveContainer" containerID="2e00571e81cdefa9efafff44c0ae2b4830651376f610bb24749fd6aa29e8a670" Nov 03 20:58:48.289452 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.289427 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-token\" (UniqueName: \"kubernetes.io/projected/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-token\") pod \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " Nov 03 20:58:48.289676 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.289497 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-data\") pod \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " Nov 03 20:58:48.289676 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.289535 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-credential-socket\") pod \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " Nov 03 20:58:48.289676 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.289597 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-podinfo\") pod \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " Nov 03 20:58:48.289676 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.289662 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istiod-ca-cert\") pod \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " Nov 03 20:58:48.290309 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.289725 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-kube-api-access-68r5x\" (UniqueName: \"kubernetes.io/projected/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-kube-api-access-68r5x\") pod \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " Nov 03 20:58:48.290562 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.290543 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-workload-certs\") pod \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " Nov 03 20:58:48.290781 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.290766 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-workload-socket\") pod \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " Nov 03 20:58:48.291179 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.289996 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-credential-socket" pod "353543b3-c2cc-4563-a2a0-ddcf5b03db1b" (UID: "353543b3-c2cc-4563-a2a0-ddcf5b03db1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.291179 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.290807 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-envoy\") pod \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\" (UID: \"353543b3-c2cc-4563-a2a0-ddcf5b03db1b\") " Nov 03 20:58:48.291765 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.291406 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:58:48.291896 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.291828 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-workload-certs" pod "353543b3-c2cc-4563-a2a0-ddcf5b03db1b" (UID: "353543b3-c2cc-4563-a2a0-ddcf5b03db1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.291972 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.291936 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-workload-socket" pod "353543b3-c2cc-4563-a2a0-ddcf5b03db1b" (UID: "353543b3-c2cc-4563-a2a0-ddcf5b03db1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.292036 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.291968 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.292284 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.292099 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-data" pod "353543b3-c2cc-4563-a2a0-ddcf5b03db1b" (UID: "353543b3-c2cc-4563-a2a0-ddcf5b03db1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.293054 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.293032 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istiod-ca-cert" pod "353543b3-c2cc-4563-a2a0-ddcf5b03db1b" (UID: "353543b3-c2cc-4563-a2a0-ddcf5b03db1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:48.298750 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.298724 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-envoy" pod "353543b3-c2cc-4563-a2a0-ddcf5b03db1b" (UID: "353543b3-c2cc-4563-a2a0-ddcf5b03db1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.299100 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.299079 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-kube-api-access-68r5x" pod "353543b3-c2cc-4563-a2a0-ddcf5b03db1b" (UID: "353543b3-c2cc-4563-a2a0-ddcf5b03db1b"). InnerVolumeSpecName "kube-api-access-68r5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:48.299285 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.299261 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-token" pod "353543b3-c2cc-4563-a2a0-ddcf5b03db1b" (UID: "353543b3-c2cc-4563-a2a0-ddcf5b03db1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:48.300746 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.300708 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-podinfo" pod "353543b3-c2cc-4563-a2a0-ddcf5b03db1b" (UID: "353543b3-c2cc-4563-a2a0-ddcf5b03db1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:48.368847 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.368818 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:58:48.371660 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.371625 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:58:48.392903 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.392865 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a092ea83-8438-4626-a3a6-0e076627ab11-istio-token\" (UniqueName: \"kubernetes.io/projected/a092ea83-8438-4626-a3a6-0e076627ab11-istio-token\") pod \"a092ea83-8438-4626-a3a6-0e076627ab11\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " Nov 03 20:58:48.393045 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.392935 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-workload-socket\") pod \"a092ea83-8438-4626-a3a6-0e076627ab11\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " Nov 03 20:58:48.393045 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.392993 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a092ea83-8438-4626-a3a6-0e076627ab11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a092ea83-8438-4626-a3a6-0e076627ab11-istiod-ca-cert\") pod \"a092ea83-8438-4626-a3a6-0e076627ab11\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " Nov 03 20:58:48.393198 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.393101 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-istio-envoy\") pod \"a092ea83-8438-4626-a3a6-0e076627ab11\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " Nov 03 20:58:48.393198 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.393133 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a092ea83-8438-4626-a3a6-0e076627ab11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a092ea83-8438-4626-a3a6-0e076627ab11-istio-podinfo\") pod \"a092ea83-8438-4626-a3a6-0e076627ab11\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " Nov 03 20:58:48.393302 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.393212 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-workload-socket" pod "a092ea83-8438-4626-a3a6-0e076627ab11" (UID: "a092ea83-8438-4626-a3a6-0e076627ab11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.393302 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.393231 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-workload-certs\") pod \"a092ea83-8438-4626-a3a6-0e076627ab11\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " Nov 03 20:58:48.393302 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.393265 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a092ea83-8438-4626-a3a6-0e076627ab11-kube-api-access-lwmtp\" (UniqueName: \"kubernetes.io/projected/a092ea83-8438-4626-a3a6-0e076627ab11-kube-api-access-lwmtp\") pod \"a092ea83-8438-4626-a3a6-0e076627ab11\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " Nov 03 20:58:48.393514 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.393320 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-credential-socket\") pod \"a092ea83-8438-4626-a3a6-0e076627ab11\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " Nov 03 20:58:48.393514 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.393370 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a092ea83-8438-4626-a3a6-0e076627ab11-istiod-ca-cert" pod "a092ea83-8438-4626-a3a6-0e076627ab11" (UID: "a092ea83-8438-4626-a3a6-0e076627ab11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:48.393628 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.393530 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-istio-data\") pod \"a092ea83-8438-4626-a3a6-0e076627ab11\" (UID: \"a092ea83-8438-4626-a3a6-0e076627ab11\") " Nov 03 20:58:48.393763 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.393657 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-workload-certs" pod "a092ea83-8438-4626-a3a6-0e076627ab11" (UID: "a092ea83-8438-4626-a3a6-0e076627ab11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.393827 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.393733 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-credential-socket" pod "a092ea83-8438-4626-a3a6-0e076627ab11" (UID: "a092ea83-8438-4626-a3a6-0e076627ab11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.393952 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.393936 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-istio-data" pod "a092ea83-8438-4626-a3a6-0e076627ab11" (UID: "a092ea83-8438-4626-a3a6-0e076627ab11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.394350 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.394308 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.394452 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.394358 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.394452 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.394373 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a092ea83-8438-4626-a3a6-0e076627ab11-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.394452 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.394387 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.394452 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.394401 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.394452 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.394414 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.394452 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.394429 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.394452 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.394442 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.394805 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.394455 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.394805 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.394469 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.394805 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.394483 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.394805 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.394496 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.394805 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.394510 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68r5x\" (UniqueName: \"kubernetes.io/projected/353543b3-c2cc-4563-a2a0-ddcf5b03db1b-kube-api-access-68r5x\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.396166 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.396135 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a092ea83-8438-4626-a3a6-0e076627ab11-istio-podinfo" pod "a092ea83-8438-4626-a3a6-0e076627ab11" (UID: "a092ea83-8438-4626-a3a6-0e076627ab11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:48.396808 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.396777 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a092ea83-8438-4626-a3a6-0e076627ab11-kube-api-access-lwmtp" pod "a092ea83-8438-4626-a3a6-0e076627ab11" (UID: "a092ea83-8438-4626-a3a6-0e076627ab11"). InnerVolumeSpecName "kube-api-access-lwmtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:48.397287 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.397256 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a092ea83-8438-4626-a3a6-0e076627ab11-istio-token" pod "a092ea83-8438-4626-a3a6-0e076627ab11" (UID: "a092ea83-8438-4626-a3a6-0e076627ab11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:48.397460 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.397374 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-istio-envoy" pod "a092ea83-8438-4626-a3a6-0e076627ab11" (UID: "a092ea83-8438-4626-a3a6-0e076627ab11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.495461 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.495324 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57aeddbf-a41f-4cc0-8f76-d589898c2144-kube-api-access-rnmh6\" (UniqueName: \"kubernetes.io/projected/57aeddbf-a41f-4cc0-8f76-d589898c2144-kube-api-access-rnmh6\") pod \"57aeddbf-a41f-4cc0-8f76-d589898c2144\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " Nov 03 20:58:48.495461 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.495431 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-envoy\") pod \"57aeddbf-a41f-4cc0-8f76-d589898c2144\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " Nov 03 20:58:48.495727 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.495484 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-credential-socket\") pod \"57aeddbf-a41f-4cc0-8f76-d589898c2144\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " Nov 03 20:58:48.495727 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.495538 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-data\") pod \"adf4f7e5-e60d-484e-823f-1a40134f48fc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " Nov 03 20:58:48.495727 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.495588 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-credential-socket\") pod \"adf4f7e5-e60d-484e-823f-1a40134f48fc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " Nov 03 20:58:48.495727 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.495681 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-workload-certs\") pod \"57aeddbf-a41f-4cc0-8f76-d589898c2144\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " Nov 03 20:58:48.496022 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.495784 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-workload-socket\") pod \"adf4f7e5-e60d-484e-823f-1a40134f48fc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " Nov 03 20:58:48.496022 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.495816 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adf4f7e5-e60d-484e-823f-1a40134f48fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adf4f7e5-e60d-484e-823f-1a40134f48fc-istiod-ca-cert\") pod \"adf4f7e5-e60d-484e-823f-1a40134f48fc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " Nov 03 20:58:48.496022 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.495845 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-envoy\") pod \"adf4f7e5-e60d-484e-823f-1a40134f48fc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " Nov 03 20:58:48.496022 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.495884 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adf4f7e5-e60d-484e-823f-1a40134f48fc-kube-api-access-bnjrm\" (UniqueName: \"kubernetes.io/projected/adf4f7e5-e60d-484e-823f-1a40134f48fc-kube-api-access-bnjrm\") pod \"adf4f7e5-e60d-484e-823f-1a40134f48fc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " Nov 03 20:58:48.496022 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.495924 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-data" pod "adf4f7e5-e60d-484e-823f-1a40134f48fc" (UID: "adf4f7e5-e60d-484e-823f-1a40134f48fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.496022 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.495937 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57aeddbf-a41f-4cc0-8f76-d589898c2144-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57aeddbf-a41f-4cc0-8f76-d589898c2144-istiod-ca-cert\") pod \"57aeddbf-a41f-4cc0-8f76-d589898c2144\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " Nov 03 20:58:48.496350 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.496093 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-workload-socket" pod "adf4f7e5-e60d-484e-823f-1a40134f48fc" (UID: "adf4f7e5-e60d-484e-823f-1a40134f48fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.496350 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.496267 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57aeddbf-a41f-4cc0-8f76-d589898c2144-istiod-ca-cert" pod "57aeddbf-a41f-4cc0-8f76-d589898c2144" (UID: "57aeddbf-a41f-4cc0-8f76-d589898c2144"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:48.496472 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.496371 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-credential-socket" pod "adf4f7e5-e60d-484e-823f-1a40134f48fc" (UID: "adf4f7e5-e60d-484e-823f-1a40134f48fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.496584 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.496561 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-credential-socket" pod "57aeddbf-a41f-4cc0-8f76-d589898c2144" (UID: "57aeddbf-a41f-4cc0-8f76-d589898c2144"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.496584 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.496574 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-workload-certs" pod "57aeddbf-a41f-4cc0-8f76-d589898c2144" (UID: "57aeddbf-a41f-4cc0-8f76-d589898c2144"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.496723 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.496697 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adf4f7e5-e60d-484e-823f-1a40134f48fc-istiod-ca-cert" pod "adf4f7e5-e60d-484e-823f-1a40134f48fc" (UID: "adf4f7e5-e60d-484e-823f-1a40134f48fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:48.496849 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.496829 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-workload-certs\") pod \"adf4f7e5-e60d-484e-823f-1a40134f48fc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " Nov 03 20:58:48.497283 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.496868 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-podinfo\") pod \"adf4f7e5-e60d-484e-823f-1a40134f48fc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " Nov 03 20:58:48.497283 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.496957 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-podinfo\") pod \"57aeddbf-a41f-4cc0-8f76-d589898c2144\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " Nov 03 20:58:48.497283 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.497129 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-token\" (UniqueName: \"kubernetes.io/projected/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-token\") pod \"57aeddbf-a41f-4cc0-8f76-d589898c2144\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " Nov 03 20:58:48.497283 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.497223 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-workload-socket\") pod \"57aeddbf-a41f-4cc0-8f76-d589898c2144\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " Nov 03 20:58:48.497283 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.497232 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-workload-certs" pod "adf4f7e5-e60d-484e-823f-1a40134f48fc" (UID: "adf4f7e5-e60d-484e-823f-1a40134f48fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.497283 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.497251 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-data\") pod \"57aeddbf-a41f-4cc0-8f76-d589898c2144\" (UID: \"57aeddbf-a41f-4cc0-8f76-d589898c2144\") " Nov 03 20:58:48.497703 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.497295 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-token\" (UniqueName: \"kubernetes.io/projected/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-token\") pod \"adf4f7e5-e60d-484e-823f-1a40134f48fc\" (UID: \"adf4f7e5-e60d-484e-823f-1a40134f48fc\") " Nov 03 20:58:48.497765 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.497736 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-workload-socket" pod "57aeddbf-a41f-4cc0-8f76-d589898c2144" (UID: "57aeddbf-a41f-4cc0-8f76-d589898c2144"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.498241 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.497938 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-data" pod "57aeddbf-a41f-4cc0-8f76-d589898c2144" (UID: "57aeddbf-a41f-4cc0-8f76-d589898c2144"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.498241 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.498213 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57aeddbf-a41f-4cc0-8f76-d589898c2144-kube-api-access-rnmh6" pod "57aeddbf-a41f-4cc0-8f76-d589898c2144" (UID: "57aeddbf-a41f-4cc0-8f76-d589898c2144"). InnerVolumeSpecName "kube-api-access-rnmh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:48.498460 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.498278 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a092ea83-8438-4626-a3a6-0e076627ab11-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.498460 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.498302 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a092ea83-8438-4626-a3a6-0e076627ab11-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.498460 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.498319 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.498460 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.498353 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.498460 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.498368 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwmtp\" (UniqueName: \"kubernetes.io/projected/a092ea83-8438-4626-a3a6-0e076627ab11-kube-api-access-lwmtp\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.498460 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.498385 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.498460 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.498399 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.498460 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.498414 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.498460 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.498430 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.498460 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.498445 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.498460 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.498458 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adf4f7e5-e60d-484e-823f-1a40134f48fc-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.499016 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.498472 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57aeddbf-a41f-4cc0-8f76-d589898c2144-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.499016 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.498485 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a092ea83-8438-4626-a3a6-0e076627ab11-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.499016 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.498499 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.499653 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.499615 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-envoy" pod "57aeddbf-a41f-4cc0-8f76-d589898c2144" (UID: "57aeddbf-a41f-4cc0-8f76-d589898c2144"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.500009 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.499975 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-envoy" pod "adf4f7e5-e60d-484e-823f-1a40134f48fc" (UID: "adf4f7e5-e60d-484e-823f-1a40134f48fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.500562 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.500538 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-podinfo" pod "57aeddbf-a41f-4cc0-8f76-d589898c2144" (UID: "57aeddbf-a41f-4cc0-8f76-d589898c2144"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:48.500798 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.500761 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-token" pod "57aeddbf-a41f-4cc0-8f76-d589898c2144" (UID: "57aeddbf-a41f-4cc0-8f76-d589898c2144"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:48.501601 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.501568 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adf4f7e5-e60d-484e-823f-1a40134f48fc-kube-api-access-bnjrm" pod "adf4f7e5-e60d-484e-823f-1a40134f48fc" (UID: "adf4f7e5-e60d-484e-823f-1a40134f48fc"). InnerVolumeSpecName "kube-api-access-bnjrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:48.501812 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.501790 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-token" pod "adf4f7e5-e60d-484e-823f-1a40134f48fc" (UID: "adf4f7e5-e60d-484e-823f-1a40134f48fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:48.502160 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.502131 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-podinfo" pod "adf4f7e5-e60d-484e-823f-1a40134f48fc" (UID: "adf4f7e5-e60d-484e-823f-1a40134f48fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:48.600234 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.600198 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.600234 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.600228 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.600234 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.600237 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.600482 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.600247 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.600482 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.600256 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnmh6\" (UniqueName: \"kubernetes.io/projected/57aeddbf-a41f-4cc0-8f76-d589898c2144-kube-api-access-rnmh6\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.600482 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.600265 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57aeddbf-a41f-4cc0-8f76-d589898c2144-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.600482 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.600275 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adf4f7e5-e60d-484e-823f-1a40134f48fc-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.600482 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.600284 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnjrm\" (UniqueName: \"kubernetes.io/projected/adf4f7e5-e60d-484e-823f-1a40134f48fc-kube-api-access-bnjrm\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.856217 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.856188 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq"] Nov 03 20:58:48.860736 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.860711 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-tsfdq"] Nov 03 20:58:48.872562 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.872542 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 20:58:48.874619 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:48.874598 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 20:58:49.225790 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:49.225709 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" Nov 03 20:58:49.225790 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:49.225728 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g" event={"ID":"a092ea83-8438-4626-a3a6-0e076627ab11","Type":"ContainerDied","Data":"6e8535d840ddaf472387e12e37ffff68011daac3337b187c04ad5d7955cabc7d"} Nov 03 20:58:49.225790 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:49.225787 4493 scope.go:122] "RemoveContainer" containerID="9f89a58d3ee4de632bbd5db4bd9c9860bade2b236b687e01d2baa7636084ba27" Nov 03 20:58:49.227280 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:49.227258 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" event={"ID":"adf4f7e5-e60d-484e-823f-1a40134f48fc","Type":"ContainerDied","Data":"3f5aebbcc16bc934bd23da2036e80a3cfc5c789bb2d81314daf55db910376436"} Nov 03 20:58:49.227280 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:49.227273 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc" Nov 03 20:58:49.228969 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:49.228951 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" Nov 03 20:58:49.229056 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:49.228950 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b" event={"ID":"57aeddbf-a41f-4cc0-8f76-d589898c2144","Type":"ContainerDied","Data":"b7c42e2b74a7f5722673a70fc0b732c51656ea1e475291793e9f6633a7a83670"} Nov 03 20:58:49.239297 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:49.239280 4493 scope.go:122] "RemoveContainer" containerID="6582c21ef2abd9d2ddab5c8f0f227b8a2355fbcb1cdea4d67c905f91f33452ea" Nov 03 20:58:49.251656 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:49.251626 4493 scope.go:122] "RemoveContainer" containerID="9571d5e8003ba1be713011a8d27c1a7b39ddd944bbc126a134c01dff54c10474" Nov 03 20:58:49.383118 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:49.383088 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc"] Nov 03 20:58:49.386677 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:49.386653 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-bbmfc"] Nov 03 20:58:49.488978 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:49.488895 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b"] Nov 03 20:58:49.492478 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:49.492452 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-9xk6b"] Nov 03 20:58:49.595090 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:49.595059 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g"] Nov 03 20:58:49.599261 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:49.599240 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-g7q2g"] Nov 03 20:58:50.144483 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:50.144455 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g" Nov 03 20:58:50.144652 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:50.144521 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb" Nov 03 20:58:50.564806 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:50.564770 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="353543b3-c2cc-4563-a2a0-ddcf5b03db1b" path="/var/lib/kubelet/pods/353543b3-c2cc-4563-a2a0-ddcf5b03db1b/volumes" Nov 03 20:58:50.565536 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:50.565516 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57aeddbf-a41f-4cc0-8f76-d589898c2144" path="/var/lib/kubelet/pods/57aeddbf-a41f-4cc0-8f76-d589898c2144/volumes" Nov 03 20:58:50.566201 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:50.566184 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a092ea83-8438-4626-a3a6-0e076627ab11" path="/var/lib/kubelet/pods/a092ea83-8438-4626-a3a6-0e076627ab11/volumes" Nov 03 20:58:50.567294 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:50.567273 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adf4f7e5-e60d-484e-823f-1a40134f48fc" path="/var/lib/kubelet/pods/adf4f7e5-e60d-484e-823f-1a40134f48fc/volumes" Nov 03 20:58:51.745985 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:58:51.745947 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:58:55.184691 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:55.184653 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt" Nov 03 20:58:55.185248 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:58:55.184774 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj" Nov 03 20:59:21.747113 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:59:21.747072 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:59:24.560962 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:24.560930 4493 generic.go:356] "Generic (PLEG): container finished" podID="701f971e-5fbb-4f4b-81ec-1dfa66005128" containerID="73764b575f50c5e1e9b4d1ff2f3b97c84f749f0cfe430d47e02c385ea31812a8" exitCode=0 Nov 03 20:59:24.562407 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:24.562383 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" event={"ID":"701f971e-5fbb-4f4b-81ec-1dfa66005128","Type":"ContainerDied","Data":"73764b575f50c5e1e9b4d1ff2f3b97c84f749f0cfe430d47e02c385ea31812a8"} Nov 03 20:59:24.562834 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:24.562819 4493 scope.go:122] "RemoveContainer" containerID="73764b575f50c5e1e9b4d1ff2f3b97c84f749f0cfe430d47e02c385ea31812a8" Nov 03 20:59:24.564612 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:24.564588 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:59:24.564689 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:24.564653 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:59:24.564726 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:24.564687 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:59:24.717671 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:24.717640 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn"] Nov 03 20:59:24.717888 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:24.717864 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" podUID="0d81ee18-7c91-43c5-b17c-4c42835b431b" containerName="istio-proxy" containerID="cri-o://971ddc644f48045a94c34fc754b920bb806f6408dbda32d15734654acf8deb04" gracePeriod=30 Nov 03 20:59:24.721998 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:24.721968 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn"] Nov 03 20:59:24.722196 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:24.722174 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" podUID="0df130a3-4a40-4837-8211-cfc0ed20d0c2" containerName="istio-proxy" containerID="cri-o://61104a1295d86f46df6e7905192f0a215308438f5977ac4efff64b6ace6de397" gracePeriod=30 Nov 03 20:59:24.764787 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:24.764759 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w"] Nov 03 20:59:24.764986 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:24.764966 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" podUID="f51d926d-5ce9-48da-8444-3042b674b606" containerName="istio-proxy" containerID="cri-o://d4dadbd01850a696b7a4235c87a0c0ef4302b254c296c21a0ed4ff2ed0f2af11" gracePeriod=30 Nov 03 20:59:24.768104 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:24.768077 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2"] Nov 03 20:59:24.768309 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:24.768289 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" podUID="1285885e-a374-4d7c-8109-0192cedd4fb2" containerName="istio-proxy" containerID="cri-o://2c3ff3ce94670ea937e47a72d79c495fb5e343a5e9b7e1d8d4f0bd38a8af7a0a" gracePeriod=30 Nov 03 20:59:25.569169 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:25.569140 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" event={"ID":"701f971e-5fbb-4f4b-81ec-1dfa66005128","Type":"ContainerStarted","Data":"92cc2bacbc8339474b32c083db15dd64c6d0de787073729b42004110badb9e15"} Nov 03 20:59:26.459931 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:26.459896 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:59:26.461420 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:26.461392 4493 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:59:26.461556 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:26.461442 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" podUID="701f971e-5fbb-4f4b-81ec-1dfa66005128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 20:59:27.459491 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:27.459455 4493 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:59:27.459965 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:27.459515 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" podUID="701f971e-5fbb-4f4b-81ec-1dfa66005128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 20:59:28.056012 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:28.055981 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j"] Nov 03 20:59:28.056235 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:28.056198 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" podUID="28a8231a-a95b-49e6-9c78-d872d17fd6cb" containerName="istio-proxy" containerID="cri-o://1a8386cb970ec6728645b685d0ea191d47f14f9ded64e64ddaa9d145a0079655" gracePeriod=30 Nov 03 20:59:28.460068 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:28.459989 4493 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:59:28.460068 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:28.460041 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" podUID="701f971e-5fbb-4f4b-81ec-1dfa66005128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 20:59:29.152041 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:29.152006 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:59:29.152229 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:29.152060 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" podUID="f51d926d-5ce9-48da-8444-3042b674b606" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:59:29.459740 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:29.459658 4493 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:59:29.459740 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:29.459720 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" podUID="701f971e-5fbb-4f4b-81ec-1dfa66005128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 20:59:30.165276 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.165248 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:59:30.174231 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.174202 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-data\") pod \"0d81ee18-7c91-43c5-b17c-4c42835b431b\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " Nov 03 20:59:30.174429 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.174289 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-credential-socket\") pod \"0d81ee18-7c91-43c5-b17c-4c42835b431b\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " Nov 03 20:59:30.174429 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.174370 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-podinfo\") pod \"0d81ee18-7c91-43c5-b17c-4c42835b431b\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " Nov 03 20:59:30.174580 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.174464 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d81ee18-7c91-43c5-b17c-4c42835b431b-kube-api-access-rpsfn\" (UniqueName: \"kubernetes.io/projected/0d81ee18-7c91-43c5-b17c-4c42835b431b-kube-api-access-rpsfn\") pod \"0d81ee18-7c91-43c5-b17c-4c42835b431b\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " Nov 03 20:59:30.174580 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.174518 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d81ee18-7c91-43c5-b17c-4c42835b431b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d81ee18-7c91-43c5-b17c-4c42835b431b-istiod-ca-cert\") pod \"0d81ee18-7c91-43c5-b17c-4c42835b431b\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " Nov 03 20:59:30.174712 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.174613 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-workload-socket\") pod \"0d81ee18-7c91-43c5-b17c-4c42835b431b\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " Nov 03 20:59:30.174712 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.174658 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-envoy\") pod \"0d81ee18-7c91-43c5-b17c-4c42835b431b\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " Nov 03 20:59:30.174712 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.174664 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-credential-socket" pod "0d81ee18-7c91-43c5-b17c-4c42835b431b" (UID: "0d81ee18-7c91-43c5-b17c-4c42835b431b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:30.174712 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.174706 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-workload-certs\") pod \"0d81ee18-7c91-43c5-b17c-4c42835b431b\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " Nov 03 20:59:30.174914 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.174748 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-data" pod "0d81ee18-7c91-43c5-b17c-4c42835b431b" (UID: "0d81ee18-7c91-43c5-b17c-4c42835b431b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:30.174914 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.174777 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-token\" (UniqueName: \"kubernetes.io/projected/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-token\") pod \"0d81ee18-7c91-43c5-b17c-4c42835b431b\" (UID: \"0d81ee18-7c91-43c5-b17c-4c42835b431b\") " Nov 03 20:59:30.175115 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.175086 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d81ee18-7c91-43c5-b17c-4c42835b431b-istiod-ca-cert" pod "0d81ee18-7c91-43c5-b17c-4c42835b431b" (UID: "0d81ee18-7c91-43c5-b17c-4c42835b431b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:30.175199 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.175108 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-workload-socket" pod "0d81ee18-7c91-43c5-b17c-4c42835b431b" (UID: "0d81ee18-7c91-43c5-b17c-4c42835b431b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:30.175541 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.175517 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-workload-certs" pod "0d81ee18-7c91-43c5-b17c-4c42835b431b" (UID: "0d81ee18-7c91-43c5-b17c-4c42835b431b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:30.175748 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.175731 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d81ee18-7c91-43c5-b17c-4c42835b431b-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.175800 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.175752 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.175800 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.175767 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.175800 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.175781 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.175800 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.175795 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.179502 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.179464 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d81ee18-7c91-43c5-b17c-4c42835b431b-kube-api-access-rpsfn" pod "0d81ee18-7c91-43c5-b17c-4c42835b431b" (UID: "0d81ee18-7c91-43c5-b17c-4c42835b431b"). InnerVolumeSpecName "kube-api-access-rpsfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:30.179901 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.179870 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-token" pod "0d81ee18-7c91-43c5-b17c-4c42835b431b" (UID: "0d81ee18-7c91-43c5-b17c-4c42835b431b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:30.181134 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.181108 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-envoy" pod "0d81ee18-7c91-43c5-b17c-4c42835b431b" (UID: "0d81ee18-7c91-43c5-b17c-4c42835b431b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:30.182503 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.182474 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-podinfo" pod "0d81ee18-7c91-43c5-b17c-4c42835b431b" (UID: "0d81ee18-7c91-43c5-b17c-4c42835b431b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:59:30.234678 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.234652 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:59:30.236815 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.236792 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:59:30.276970 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.276939 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-workload-socket\") pod \"1285885e-a374-4d7c-8109-0192cedd4fb2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " Nov 03 20:59:30.276970 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.276976 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-token\" (UniqueName: \"kubernetes.io/projected/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-token\") pod \"1285885e-a374-4d7c-8109-0192cedd4fb2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " Nov 03 20:59:30.277186 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.277015 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-podinfo\") pod \"1285885e-a374-4d7c-8109-0192cedd4fb2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " Nov 03 20:59:30.277186 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.277036 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1285885e-a374-4d7c-8109-0192cedd4fb2-kube-api-access-btl62\" (UniqueName: \"kubernetes.io/projected/1285885e-a374-4d7c-8109-0192cedd4fb2-kube-api-access-btl62\") pod \"1285885e-a374-4d7c-8109-0192cedd4fb2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " Nov 03 20:59:30.277186 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.277080 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-istio-envoy\") pod \"f51d926d-5ce9-48da-8444-3042b674b606\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " Nov 03 20:59:30.277186 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.277115 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-credential-socket\") pod \"f51d926d-5ce9-48da-8444-3042b674b606\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " Nov 03 20:59:30.277186 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.277143 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1285885e-a374-4d7c-8109-0192cedd4fb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1285885e-a374-4d7c-8109-0192cedd4fb2-istiod-ca-cert\") pod \"1285885e-a374-4d7c-8109-0192cedd4fb2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " Nov 03 20:59:30.277186 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.277160 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-workload-certs\") pod \"1285885e-a374-4d7c-8109-0192cedd4fb2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " Nov 03 20:59:30.277186 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.277177 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f51d926d-5ce9-48da-8444-3042b674b606-kube-api-access-j4hgn\" (UniqueName: \"kubernetes.io/projected/f51d926d-5ce9-48da-8444-3042b674b606-kube-api-access-j4hgn\") pod \"f51d926d-5ce9-48da-8444-3042b674b606\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " Nov 03 20:59:30.277646 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.277201 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-istio-data\") pod \"f51d926d-5ce9-48da-8444-3042b674b606\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " Nov 03 20:59:30.277646 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.277286 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-workload-certs\") pod \"f51d926d-5ce9-48da-8444-3042b674b606\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " Nov 03 20:59:30.277646 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.277309 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f51d926d-5ce9-48da-8444-3042b674b606-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51d926d-5ce9-48da-8444-3042b674b606-istio-podinfo\") pod \"f51d926d-5ce9-48da-8444-3042b674b606\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " Nov 03 20:59:30.277646 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.277362 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-workload-socket\") pod \"f51d926d-5ce9-48da-8444-3042b674b606\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " Nov 03 20:59:30.277646 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.277398 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f51d926d-5ce9-48da-8444-3042b674b606-istio-token\" (UniqueName: \"kubernetes.io/projected/f51d926d-5ce9-48da-8444-3042b674b606-istio-token\") pod \"f51d926d-5ce9-48da-8444-3042b674b606\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " Nov 03 20:59:30.277646 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.277415 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f51d926d-5ce9-48da-8444-3042b674b606-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51d926d-5ce9-48da-8444-3042b674b606-istiod-ca-cert\") pod \"f51d926d-5ce9-48da-8444-3042b674b606\" (UID: \"f51d926d-5ce9-48da-8444-3042b674b606\") " Nov 03 20:59:30.277646 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.277458 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-credential-socket\") pod \"1285885e-a374-4d7c-8109-0192cedd4fb2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " Nov 03 20:59:30.278014 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.277738 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-envoy\") pod \"1285885e-a374-4d7c-8109-0192cedd4fb2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " Nov 03 20:59:30.278014 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.277828 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-data\") pod \"1285885e-a374-4d7c-8109-0192cedd4fb2\" (UID: \"1285885e-a374-4d7c-8109-0192cedd4fb2\") " Nov 03 20:59:30.278127 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.278106 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-istio-data" pod "f51d926d-5ce9-48da-8444-3042b674b606" (UID: "f51d926d-5ce9-48da-8444-3042b674b606"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:30.278531 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.278285 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-workload-certs" pod "1285885e-a374-4d7c-8109-0192cedd4fb2" (UID: "1285885e-a374-4d7c-8109-0192cedd4fb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:30.278660 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.278570 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-data" pod "1285885e-a374-4d7c-8109-0192cedd4fb2" (UID: "1285885e-a374-4d7c-8109-0192cedd4fb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:30.278993 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.278785 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-workload-certs" pod "f51d926d-5ce9-48da-8444-3042b674b606" (UID: "f51d926d-5ce9-48da-8444-3042b674b606"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:30.278993 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.278801 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-credential-socket" pod "f51d926d-5ce9-48da-8444-3042b674b606" (UID: "f51d926d-5ce9-48da-8444-3042b674b606"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:30.278993 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.278850 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.278993 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.278873 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.278993 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.278888 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.278993 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.278902 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.278993 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.278917 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.278993 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.278935 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d81ee18-7c91-43c5-b17c-4c42835b431b-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.278993 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.278950 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpsfn\" (UniqueName: \"kubernetes.io/projected/0d81ee18-7c91-43c5-b17c-4c42835b431b-kube-api-access-rpsfn\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.279638 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.279200 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-workload-socket" pod "f51d926d-5ce9-48da-8444-3042b674b606" (UID: "f51d926d-5ce9-48da-8444-3042b674b606"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:30.279638 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.279464 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1285885e-a374-4d7c-8109-0192cedd4fb2-istiod-ca-cert" pod "1285885e-a374-4d7c-8109-0192cedd4fb2" (UID: "1285885e-a374-4d7c-8109-0192cedd4fb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:30.279744 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.279712 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-credential-socket" pod "1285885e-a374-4d7c-8109-0192cedd4fb2" (UID: "1285885e-a374-4d7c-8109-0192cedd4fb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:30.279895 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.279878 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-workload-socket" pod "1285885e-a374-4d7c-8109-0192cedd4fb2" (UID: "1285885e-a374-4d7c-8109-0192cedd4fb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:30.279895 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.279877 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f51d926d-5ce9-48da-8444-3042b674b606-istiod-ca-cert" pod "f51d926d-5ce9-48da-8444-3042b674b606" (UID: "f51d926d-5ce9-48da-8444-3042b674b606"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:30.283061 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.283027 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51d926d-5ce9-48da-8444-3042b674b606-istio-token" pod "f51d926d-5ce9-48da-8444-3042b674b606" (UID: "f51d926d-5ce9-48da-8444-3042b674b606"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:30.287413 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.283277 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-podinfo" pod "1285885e-a374-4d7c-8109-0192cedd4fb2" (UID: "1285885e-a374-4d7c-8109-0192cedd4fb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:59:30.287683 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.287644 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1285885e-a374-4d7c-8109-0192cedd4fb2-kube-api-access-btl62" pod "1285885e-a374-4d7c-8109-0192cedd4fb2" (UID: "1285885e-a374-4d7c-8109-0192cedd4fb2"). InnerVolumeSpecName "kube-api-access-btl62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:30.287683 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.287650 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f51d926d-5ce9-48da-8444-3042b674b606-istio-podinfo" pod "f51d926d-5ce9-48da-8444-3042b674b606" (UID: "f51d926d-5ce9-48da-8444-3042b674b606"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:59:30.287834 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.287737 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-envoy" pod "1285885e-a374-4d7c-8109-0192cedd4fb2" (UID: "1285885e-a374-4d7c-8109-0192cedd4fb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:30.288284 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.288241 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-token" pod "1285885e-a374-4d7c-8109-0192cedd4fb2" (UID: "1285885e-a374-4d7c-8109-0192cedd4fb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:30.296960 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.296900 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-istio-envoy" pod "f51d926d-5ce9-48da-8444-3042b674b606" (UID: "f51d926d-5ce9-48da-8444-3042b674b606"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:30.304355 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.303609 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51d926d-5ce9-48da-8444-3042b674b606-kube-api-access-j4hgn" pod "f51d926d-5ce9-48da-8444-3042b674b606" (UID: "f51d926d-5ce9-48da-8444-3042b674b606"). InnerVolumeSpecName "kube-api-access-j4hgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:30.380240 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.380206 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.380240 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.380238 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.380240 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.380247 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.380557 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.380256 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btl62\" (UniqueName: \"kubernetes.io/projected/1285885e-a374-4d7c-8109-0192cedd4fb2-kube-api-access-btl62\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.380557 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.380267 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.380557 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.380276 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.380557 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.380283 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1285885e-a374-4d7c-8109-0192cedd4fb2-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.380557 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.380292 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4hgn\" (UniqueName: \"kubernetes.io/projected/f51d926d-5ce9-48da-8444-3042b674b606-kube-api-access-j4hgn\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.380557 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.380303 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.380557 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.380311 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51d926d-5ce9-48da-8444-3042b674b606-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.380557 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.380319 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51d926d-5ce9-48da-8444-3042b674b606-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.380557 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.380344 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51d926d-5ce9-48da-8444-3042b674b606-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.380557 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.380356 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51d926d-5ce9-48da-8444-3042b674b606-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.380557 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.380365 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.380557 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.380373 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1285885e-a374-4d7c-8109-0192cedd4fb2-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.459470 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.459435 4493 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:59:30.460524 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.460166 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" podUID="701f971e-5fbb-4f4b-81ec-1dfa66005128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 20:59:30.474706 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.474679 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:59:30.583008 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.582925 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-workload-socket\") pod \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " Nov 03 20:59:30.583161 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.583052 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-workload-certs\") pod \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " Nov 03 20:59:30.583161 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.583094 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-data\") pod \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " Nov 03 20:59:30.583161 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.583131 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-envoy\") pod \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " Nov 03 20:59:30.583391 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.583218 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-credential-socket\") pod \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " Nov 03 20:59:30.583391 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.583256 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istiod-ca-cert\") pod \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " Nov 03 20:59:30.583391 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.583306 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0df130a3-4a40-4837-8211-cfc0ed20d0c2-kube-api-access-8rxpv\" (UniqueName: \"kubernetes.io/projected/0df130a3-4a40-4837-8211-cfc0ed20d0c2-kube-api-access-8rxpv\") pod \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " Nov 03 20:59:30.583391 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.583373 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-token\" (UniqueName: \"kubernetes.io/projected/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-token\") pod \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " Nov 03 20:59:30.583613 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.583417 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-podinfo\") pod \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\" (UID: \"0df130a3-4a40-4837-8211-cfc0ed20d0c2\") " Nov 03 20:59:30.614923 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.612752 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-workload-certs" pod "0df130a3-4a40-4837-8211-cfc0ed20d0c2" (UID: "0df130a3-4a40-4837-8211-cfc0ed20d0c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:30.614923 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.612972 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-workload-socket" pod "0df130a3-4a40-4837-8211-cfc0ed20d0c2" (UID: "0df130a3-4a40-4837-8211-cfc0ed20d0c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:30.614923 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.613053 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istiod-ca-cert" pod "0df130a3-4a40-4837-8211-cfc0ed20d0c2" (UID: "0df130a3-4a40-4837-8211-cfc0ed20d0c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:30.614923 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.613454 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-data" pod "0df130a3-4a40-4837-8211-cfc0ed20d0c2" (UID: "0df130a3-4a40-4837-8211-cfc0ed20d0c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:30.614923 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.613862 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-credential-socket" pod "0df130a3-4a40-4837-8211-cfc0ed20d0c2" (UID: "0df130a3-4a40-4837-8211-cfc0ed20d0c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:30.617852 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.617826 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-envoy" pod "0df130a3-4a40-4837-8211-cfc0ed20d0c2" (UID: "0df130a3-4a40-4837-8211-cfc0ed20d0c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:30.617964 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.617869 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-token" pod "0df130a3-4a40-4837-8211-cfc0ed20d0c2" (UID: "0df130a3-4a40-4837-8211-cfc0ed20d0c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:30.617964 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.617927 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-podinfo" pod "0df130a3-4a40-4837-8211-cfc0ed20d0c2" (UID: "0df130a3-4a40-4837-8211-cfc0ed20d0c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:59:30.619169 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.619146 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df130a3-4a40-4837-8211-cfc0ed20d0c2-kube-api-access-8rxpv" pod "0df130a3-4a40-4837-8211-cfc0ed20d0c2" (UID: "0df130a3-4a40-4837-8211-cfc0ed20d0c2"). InnerVolumeSpecName "kube-api-access-8rxpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:30.641349 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.637957 4493 generic.go:356] "Generic (PLEG): container finished" podID="0df130a3-4a40-4837-8211-cfc0ed20d0c2" containerID="61104a1295d86f46df6e7905192f0a215308438f5977ac4efff64b6ace6de397" exitCode=0 Nov 03 20:59:30.641349 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.638062 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" event={"ID":"0df130a3-4a40-4837-8211-cfc0ed20d0c2","Type":"ContainerDied","Data":"61104a1295d86f46df6e7905192f0a215308438f5977ac4efff64b6ace6de397"} Nov 03 20:59:30.641349 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.638111 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" event={"ID":"0df130a3-4a40-4837-8211-cfc0ed20d0c2","Type":"ContainerDied","Data":"fcbd1a2f594c0716edf000a49e66b4ecb524ef7ece30029c96c2b9c37ddd253d"} Nov 03 20:59:30.641349 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.638140 4493 scope.go:122] "RemoveContainer" containerID="61104a1295d86f46df6e7905192f0a215308438f5977ac4efff64b6ace6de397" Nov 03 20:59:30.641349 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.638290 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn" Nov 03 20:59:30.645636 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.645262 4493 generic.go:356] "Generic (PLEG): container finished" podID="1285885e-a374-4d7c-8109-0192cedd4fb2" containerID="2c3ff3ce94670ea937e47a72d79c495fb5e343a5e9b7e1d8d4f0bd38a8af7a0a" exitCode=0 Nov 03 20:59:30.645636 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.645326 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" event={"ID":"1285885e-a374-4d7c-8109-0192cedd4fb2","Type":"ContainerDied","Data":"2c3ff3ce94670ea937e47a72d79c495fb5e343a5e9b7e1d8d4f0bd38a8af7a0a"} Nov 03 20:59:30.645636 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.645326 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" Nov 03 20:59:30.645636 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.645393 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2" event={"ID":"1285885e-a374-4d7c-8109-0192cedd4fb2","Type":"ContainerDied","Data":"1a9df3b922ff1ee91db6b52ca34b1dd61be077888431483023f8505b1adb4922"} Nov 03 20:59:30.647285 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.647267 4493 generic.go:356] "Generic (PLEG): container finished" podID="0d81ee18-7c91-43c5-b17c-4c42835b431b" containerID="971ddc644f48045a94c34fc754b920bb806f6408dbda32d15734654acf8deb04" exitCode=0 Nov 03 20:59:30.647405 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.647372 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" Nov 03 20:59:30.647405 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.647382 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" event={"ID":"0d81ee18-7c91-43c5-b17c-4c42835b431b","Type":"ContainerDied","Data":"971ddc644f48045a94c34fc754b920bb806f6408dbda32d15734654acf8deb04"} Nov 03 20:59:30.647603 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.647413 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn" event={"ID":"0d81ee18-7c91-43c5-b17c-4c42835b431b","Type":"ContainerDied","Data":"eb79a0ea2963dd1e8fc8c6ae01cc7df2ed617c011d09eec3eaa179fb7d14606b"} Nov 03 20:59:30.649124 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.649103 4493 generic.go:356] "Generic (PLEG): container finished" podID="f51d926d-5ce9-48da-8444-3042b674b606" containerID="d4dadbd01850a696b7a4235c87a0c0ef4302b254c296c21a0ed4ff2ed0f2af11" exitCode=0 Nov 03 20:59:30.649210 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.649132 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" event={"ID":"f51d926d-5ce9-48da-8444-3042b674b606","Type":"ContainerDied","Data":"d4dadbd01850a696b7a4235c87a0c0ef4302b254c296c21a0ed4ff2ed0f2af11"} Nov 03 20:59:30.649210 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.649168 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" event={"ID":"f51d926d-5ce9-48da-8444-3042b674b606","Type":"ContainerDied","Data":"ef9d3256b52dc5492babd7ce3a5feabac82b0ce6befab532e0ebb9f728b8613a"} Nov 03 20:59:30.649210 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.649182 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w" Nov 03 20:59:30.655755 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.655734 4493 scope.go:122] "RemoveContainer" containerID="61104a1295d86f46df6e7905192f0a215308438f5977ac4efff64b6ace6de397" Nov 03 20:59:30.656017 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:59:30.655993 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61104a1295d86f46df6e7905192f0a215308438f5977ac4efff64b6ace6de397\": container with ID starting with 61104a1295d86f46df6e7905192f0a215308438f5977ac4efff64b6ace6de397 not found: ID does not exist" containerID="61104a1295d86f46df6e7905192f0a215308438f5977ac4efff64b6ace6de397" Nov 03 20:59:30.656101 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.656017 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61104a1295d86f46df6e7905192f0a215308438f5977ac4efff64b6ace6de397"} err="failed to get container status \"61104a1295d86f46df6e7905192f0a215308438f5977ac4efff64b6ace6de397\": rpc error: code = NotFound desc = could not find container \"61104a1295d86f46df6e7905192f0a215308438f5977ac4efff64b6ace6de397\": container with ID starting with 61104a1295d86f46df6e7905192f0a215308438f5977ac4efff64b6ace6de397 not found: ID does not exist" Nov 03 20:59:30.656101 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.656033 4493 scope.go:122] "RemoveContainer" containerID="2c3ff3ce94670ea937e47a72d79c495fb5e343a5e9b7e1d8d4f0bd38a8af7a0a" Nov 03 20:59:30.666846 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.666061 4493 scope.go:122] "RemoveContainer" containerID="2c3ff3ce94670ea937e47a72d79c495fb5e343a5e9b7e1d8d4f0bd38a8af7a0a" Nov 03 20:59:30.666846 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.664922 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:59:30.666846 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:59:30.666558 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c3ff3ce94670ea937e47a72d79c495fb5e343a5e9b7e1d8d4f0bd38a8af7a0a\": container with ID starting with 2c3ff3ce94670ea937e47a72d79c495fb5e343a5e9b7e1d8d4f0bd38a8af7a0a not found: ID does not exist" containerID="2c3ff3ce94670ea937e47a72d79c495fb5e343a5e9b7e1d8d4f0bd38a8af7a0a" Nov 03 20:59:30.666846 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.666656 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c3ff3ce94670ea937e47a72d79c495fb5e343a5e9b7e1d8d4f0bd38a8af7a0a"} err="failed to get container status \"2c3ff3ce94670ea937e47a72d79c495fb5e343a5e9b7e1d8d4f0bd38a8af7a0a\": rpc error: code = NotFound desc = could not find container \"2c3ff3ce94670ea937e47a72d79c495fb5e343a5e9b7e1d8d4f0bd38a8af7a0a\": container with ID starting with 2c3ff3ce94670ea937e47a72d79c495fb5e343a5e9b7e1d8d4f0bd38a8af7a0a not found: ID does not exist" Nov 03 20:59:30.666846 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.666699 4493 scope.go:122] "RemoveContainer" containerID="971ddc644f48045a94c34fc754b920bb806f6408dbda32d15734654acf8deb04" Nov 03 20:59:30.709526 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.692540 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.709526 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.692574 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.709526 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.692588 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.709526 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.692603 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.709526 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.692617 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.709526 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.692631 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rxpv\" (UniqueName: \"kubernetes.io/projected/0df130a3-4a40-4837-8211-cfc0ed20d0c2-kube-api-access-8rxpv\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.709526 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.692647 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.709526 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.692660 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0df130a3-4a40-4837-8211-cfc0ed20d0c2-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.709526 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.692674 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df130a3-4a40-4837-8211-cfc0ed20d0c2-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:30.710774 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.710742 4493 scope.go:122] "RemoveContainer" containerID="971ddc644f48045a94c34fc754b920bb806f6408dbda32d15734654acf8deb04" Nov 03 20:59:30.711225 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:59:30.711196 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"971ddc644f48045a94c34fc754b920bb806f6408dbda32d15734654acf8deb04\": container with ID starting with 971ddc644f48045a94c34fc754b920bb806f6408dbda32d15734654acf8deb04 not found: ID does not exist" containerID="971ddc644f48045a94c34fc754b920bb806f6408dbda32d15734654acf8deb04" Nov 03 20:59:30.711316 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.711238 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"971ddc644f48045a94c34fc754b920bb806f6408dbda32d15734654acf8deb04"} err="failed to get container status \"971ddc644f48045a94c34fc754b920bb806f6408dbda32d15734654acf8deb04\": rpc error: code = NotFound desc = could not find container \"971ddc644f48045a94c34fc754b920bb806f6408dbda32d15734654acf8deb04\": container with ID starting with 971ddc644f48045a94c34fc754b920bb806f6408dbda32d15734654acf8deb04 not found: ID does not exist" Nov 03 20:59:30.711316 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.711268 4493 scope.go:122] "RemoveContainer" containerID="d4dadbd01850a696b7a4235c87a0c0ef4302b254c296c21a0ed4ff2ed0f2af11" Nov 03 20:59:30.730813 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.730791 4493 scope.go:122] "RemoveContainer" containerID="b258cb6acf7f1e1b407d52ef8cb5d61f055376824a5ec68e5fed5803159b2293" Nov 03 20:59:30.768563 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.768542 4493 scope.go:122] "RemoveContainer" containerID="d4dadbd01850a696b7a4235c87a0c0ef4302b254c296c21a0ed4ff2ed0f2af11" Nov 03 20:59:30.768894 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:59:30.768873 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4dadbd01850a696b7a4235c87a0c0ef4302b254c296c21a0ed4ff2ed0f2af11\": container with ID starting with d4dadbd01850a696b7a4235c87a0c0ef4302b254c296c21a0ed4ff2ed0f2af11 not found: ID does not exist" containerID="d4dadbd01850a696b7a4235c87a0c0ef4302b254c296c21a0ed4ff2ed0f2af11" Nov 03 20:59:30.768973 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.768903 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4dadbd01850a696b7a4235c87a0c0ef4302b254c296c21a0ed4ff2ed0f2af11"} err="failed to get container status \"d4dadbd01850a696b7a4235c87a0c0ef4302b254c296c21a0ed4ff2ed0f2af11\": rpc error: code = NotFound desc = could not find container \"d4dadbd01850a696b7a4235c87a0c0ef4302b254c296c21a0ed4ff2ed0f2af11\": container with ID starting with d4dadbd01850a696b7a4235c87a0c0ef4302b254c296c21a0ed4ff2ed0f2af11 not found: ID does not exist" Nov 03 20:59:30.768973 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.768927 4493 scope.go:122] "RemoveContainer" containerID="b258cb6acf7f1e1b407d52ef8cb5d61f055376824a5ec68e5fed5803159b2293" Nov 03 20:59:30.769210 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:59:30.769194 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b258cb6acf7f1e1b407d52ef8cb5d61f055376824a5ec68e5fed5803159b2293\": container with ID starting with b258cb6acf7f1e1b407d52ef8cb5d61f055376824a5ec68e5fed5803159b2293 not found: ID does not exist" containerID="b258cb6acf7f1e1b407d52ef8cb5d61f055376824a5ec68e5fed5803159b2293" Nov 03 20:59:30.769257 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.769214 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b258cb6acf7f1e1b407d52ef8cb5d61f055376824a5ec68e5fed5803159b2293"} err="failed to get container status \"b258cb6acf7f1e1b407d52ef8cb5d61f055376824a5ec68e5fed5803159b2293\": rpc error: code = NotFound desc = could not find container \"b258cb6acf7f1e1b407d52ef8cb5d61f055376824a5ec68e5fed5803159b2293\": container with ID starting with b258cb6acf7f1e1b407d52ef8cb5d61f055376824a5ec68e5fed5803159b2293 not found: ID does not exist" Nov 03 20:59:30.778859 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.778826 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w"] Nov 03 20:59:30.830961 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.830352 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qkd7w"] Nov 03 20:59:30.920659 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.919381 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2"] Nov 03 20:59:30.924054 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:30.924019 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-66nr2"] Nov 03 20:59:31.012415 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.012324 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn"] Nov 03 20:59:31.016016 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.015985 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-j6sqn"] Nov 03 20:59:31.113297 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.113261 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn"] Nov 03 20:59:31.116678 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.116650 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-xnlfn"] Nov 03 20:59:31.463890 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.463863 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:59:31.515406 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.515372 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5"] Nov 03 20:59:31.515988 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.515972 4493 cpu_manager.go:414] "containerMap: removing container" podUID="57aeddbf-a41f-4cc0-8f76-d589898c2144" containerName="istio-proxy" Nov 03 20:59:31.516070 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.515992 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57aeddbf-a41f-4cc0-8f76-d589898c2144" containerName="istio-proxy" Nov 03 20:59:31.516070 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.516014 4493 cpu_manager.go:414] "containerMap: removing container" podUID="353543b3-c2cc-4563-a2a0-ddcf5b03db1b" containerName="istio-proxy" Nov 03 20:59:31.516070 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.516019 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="353543b3-c2cc-4563-a2a0-ddcf5b03db1b" containerName="istio-proxy" Nov 03 20:59:31.516070 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.516031 4493 cpu_manager.go:414] "containerMap: removing container" podUID="a092ea83-8438-4626-a3a6-0e076627ab11" containerName="istio-proxy" Nov 03 20:59:31.516070 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.516036 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a092ea83-8438-4626-a3a6-0e076627ab11" containerName="istio-proxy" Nov 03 20:59:31.516070 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.516045 4493 cpu_manager.go:414] "containerMap: removing container" podUID="adf4f7e5-e60d-484e-823f-1a40134f48fc" containerName="istio-proxy" Nov 03 20:59:31.516070 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.516050 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adf4f7e5-e60d-484e-823f-1a40134f48fc" containerName="istio-proxy" Nov 03 20:59:31.516070 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.516065 4493 cpu_manager.go:414] "containerMap: removing container" podUID="0df130a3-4a40-4837-8211-cfc0ed20d0c2" containerName="istio-proxy" Nov 03 20:59:31.516070 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.516070 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0df130a3-4a40-4837-8211-cfc0ed20d0c2" containerName="istio-proxy" Nov 03 20:59:31.516397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.516083 4493 cpu_manager.go:414] "containerMap: removing container" podUID="0d81ee18-7c91-43c5-b17c-4c42835b431b" containerName="istio-proxy" Nov 03 20:59:31.516397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.516088 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d81ee18-7c91-43c5-b17c-4c42835b431b" containerName="istio-proxy" Nov 03 20:59:31.516397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.516097 4493 cpu_manager.go:414] "containerMap: removing container" podUID="f51d926d-5ce9-48da-8444-3042b674b606" containerName="istio-proxy" Nov 03 20:59:31.516397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.516102 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f51d926d-5ce9-48da-8444-3042b674b606" containerName="istio-proxy" Nov 03 20:59:31.516397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.516108 4493 cpu_manager.go:414] "containerMap: removing container" podUID="1285885e-a374-4d7c-8109-0192cedd4fb2" containerName="istio-proxy" Nov 03 20:59:31.516397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.516112 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1285885e-a374-4d7c-8109-0192cedd4fb2" containerName="istio-proxy" Nov 03 20:59:31.516397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.516125 4493 cpu_manager.go:414] "containerMap: removing container" podUID="f51d926d-5ce9-48da-8444-3042b674b606" containerName="istio-proxy" Nov 03 20:59:31.516397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.516129 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f51d926d-5ce9-48da-8444-3042b674b606" containerName="istio-proxy" Nov 03 20:59:31.516397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.516140 4493 cpu_manager.go:414] "containerMap: removing container" podUID="a092ea83-8438-4626-a3a6-0e076627ab11" containerName="istio-proxy" Nov 03 20:59:31.516397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.516144 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a092ea83-8438-4626-a3a6-0e076627ab11" containerName="istio-proxy" Nov 03 20:59:31.516397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.516227 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="57aeddbf-a41f-4cc0-8f76-d589898c2144" containerName="istio-proxy" Nov 03 20:59:31.516397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.516237 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="353543b3-c2cc-4563-a2a0-ddcf5b03db1b" containerName="istio-proxy" Nov 03 20:59:31.516397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.516244 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a092ea83-8438-4626-a3a6-0e076627ab11" containerName="istio-proxy" Nov 03 20:59:31.516397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.516251 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="adf4f7e5-e60d-484e-823f-1a40134f48fc" containerName="istio-proxy" Nov 03 20:59:31.516397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.516259 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0df130a3-4a40-4837-8211-cfc0ed20d0c2" containerName="istio-proxy" Nov 03 20:59:31.516397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.516267 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d81ee18-7c91-43c5-b17c-4c42835b431b" containerName="istio-proxy" Nov 03 20:59:31.516397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.516272 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f51d926d-5ce9-48da-8444-3042b674b606" containerName="istio-proxy" Nov 03 20:59:31.516397 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.516276 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1285885e-a374-4d7c-8109-0192cedd4fb2" containerName="istio-proxy" Nov 03 20:59:31.521403 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.521382 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5" Nov 03 20:59:31.570805 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.570778 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-1ec1b6-serving-cert\"" Nov 03 20:59:31.573234 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.573211 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5"] Nov 03 20:59:31.619048 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.619018 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a0808b2-7798-4f60-9435-063b72bbdb21-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5\" (UID: \"6a0808b2-7798-4f60-9435-063b72bbdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5" Nov 03 20:59:31.619196 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.619083 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a0808b2-7798-4f60-9435-063b72bbdb21-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5\" (UID: \"6a0808b2-7798-4f60-9435-063b72bbdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5" Nov 03 20:59:31.619260 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.619232 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a0808b2-7798-4f60-9435-063b72bbdb21-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5\" (UID: \"6a0808b2-7798-4f60-9435-063b72bbdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5" Nov 03 20:59:31.619405 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.619286 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzgmk\" (UniqueName: \"kubernetes.io/projected/6a0808b2-7798-4f60-9435-063b72bbdb21-kube-api-access-rzgmk\") pod \"maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5\" (UID: \"6a0808b2-7798-4f60-9435-063b72bbdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5" Nov 03 20:59:31.662830 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.662800 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 20:59:31.720634 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.720549 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzgmk\" (UniqueName: \"kubernetes.io/projected/6a0808b2-7798-4f60-9435-063b72bbdb21-kube-api-access-rzgmk\") pod \"maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5\" (UID: \"6a0808b2-7798-4f60-9435-063b72bbdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5" Nov 03 20:59:31.720822 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.720783 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a0808b2-7798-4f60-9435-063b72bbdb21-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5\" (UID: \"6a0808b2-7798-4f60-9435-063b72bbdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5" Nov 03 20:59:31.720924 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.720864 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a0808b2-7798-4f60-9435-063b72bbdb21-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5\" (UID: \"6a0808b2-7798-4f60-9435-063b72bbdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5" Nov 03 20:59:31.720924 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:59:31.720904 4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-serving-cert: secret "maas-api-e2e-shared-a-w6-1ec1b6-serving-cert" not found Nov 03 20:59:31.721045 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.720942 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a0808b2-7798-4f60-9435-063b72bbdb21-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5\" (UID: \"6a0808b2-7798-4f60-9435-063b72bbdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5" Nov 03 20:59:31.721045 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:59:31.720975 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a0808b2-7798-4f60-9435-063b72bbdb21-maas-api-tls podName:6a0808b2-7798-4f60-9435-063b72bbdb21 nodeName:}" failed. No retries permitted until 2026-11-03 20:59:32.220952377 +0000 UTC m=+1843.778942871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a0808b2-7798-4f60-9435-063b72bbdb21-maas-api-tls") pod "maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5" (UID: "6a0808b2-7798-4f60-9435-063b72bbdb21") : secret "maas-api-e2e-shared-a-w6-1ec1b6-serving-cert" not found Nov 03 20:59:31.721706 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.721623 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a0808b2-7798-4f60-9435-063b72bbdb21-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5\" (UID: \"6a0808b2-7798-4f60-9435-063b72bbdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5" Nov 03 20:59:31.723790 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.723769 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a0808b2-7798-4f60-9435-063b72bbdb21-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5\" (UID: \"6a0808b2-7798-4f60-9435-063b72bbdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5" Nov 03 20:59:31.778090 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:31.778054 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzgmk\" (UniqueName: \"kubernetes.io/projected/6a0808b2-7798-4f60-9435-063b72bbdb21-kube-api-access-rzgmk\") pod \"maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5\" (UID: \"6a0808b2-7798-4f60-9435-063b72bbdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5" Nov 03 20:59:32.231626 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:32.231570 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a0808b2-7798-4f60-9435-063b72bbdb21-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5\" (UID: \"6a0808b2-7798-4f60-9435-063b72bbdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5" Nov 03 20:59:32.234486 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:32.234456 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a0808b2-7798-4f60-9435-063b72bbdb21-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5\" (UID: \"6a0808b2-7798-4f60-9435-063b72bbdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5" Nov 03 20:59:32.434764 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:32.434730 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5" Nov 03 20:59:32.564825 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:32.564792 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d81ee18-7c91-43c5-b17c-4c42835b431b" path="/var/lib/kubelet/pods/0d81ee18-7c91-43c5-b17c-4c42835b431b/volumes" Nov 03 20:59:32.565295 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:32.565281 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0df130a3-4a40-4837-8211-cfc0ed20d0c2" path="/var/lib/kubelet/pods/0df130a3-4a40-4837-8211-cfc0ed20d0c2/volumes" Nov 03 20:59:32.565740 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:32.565726 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1285885e-a374-4d7c-8109-0192cedd4fb2" path="/var/lib/kubelet/pods/1285885e-a374-4d7c-8109-0192cedd4fb2/volumes" Nov 03 20:59:32.566513 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:32.566498 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f51d926d-5ce9-48da-8444-3042b674b606" path="/var/lib/kubelet/pods/f51d926d-5ce9-48da-8444-3042b674b606/volumes" Nov 03 20:59:32.810155 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:32.810122 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5"] Nov 03 20:59:32.815302 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:59:32.815267 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0808b2_7798_4f60_9435_063b72bbdb21.slice/crio-d656819b3d12ec61c757a0300fed5527a29363433d718728766a745f58459813 WatchSource:0}: Error finding container d656819b3d12ec61c757a0300fed5527a29363433d718728766a745f58459813: Status 404 returned error can't find the container with id d656819b3d12ec61c757a0300fed5527a29363433d718728766a745f58459813 Nov 03 20:59:32.817150 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:32.817130 4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:59:33.330561 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.330505 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:59:33.447686 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.447651 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-data\") pod \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " Nov 03 20:59:33.447876 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.447858 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-workload-socket\") pod \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " Nov 03 20:59:33.447945 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.447892 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istiod-ca-cert\") pod \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " Nov 03 20:59:33.447945 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.447928 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-workload-certs\") pod \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " Nov 03 20:59:33.447945 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.447890 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-data" pod "28a8231a-a95b-49e6-9c78-d872d17fd6cb" (UID: "28a8231a-a95b-49e6-9c78-d872d17fd6cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.448096 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.447960 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a8231a-a95b-49e6-9c78-d872d17fd6cb-kube-api-access-5b9gd\" (UniqueName: \"kubernetes.io/projected/28a8231a-a95b-49e6-9c78-d872d17fd6cb-kube-api-access-5b9gd\") pod \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " Nov 03 20:59:33.448096 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.448014 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-envoy\") pod \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " Nov 03 20:59:33.448096 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.448027 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-workload-socket" pod "28a8231a-a95b-49e6-9c78-d872d17fd6cb" (UID: "28a8231a-a95b-49e6-9c78-d872d17fd6cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.448096 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.448047 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-credential-socket\") pod \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " Nov 03 20:59:33.448287 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.448131 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-token\" (UniqueName: \"kubernetes.io/projected/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-token\") pod \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " Nov 03 20:59:33.448287 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.448207 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-workload-certs" pod "28a8231a-a95b-49e6-9c78-d872d17fd6cb" (UID: "28a8231a-a95b-49e6-9c78-d872d17fd6cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.448287 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.448276 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-podinfo\") pod \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\" (UID: \"28a8231a-a95b-49e6-9c78-d872d17fd6cb\") " Nov 03 20:59:33.448535 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.448276 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istiod-ca-cert" pod "28a8231a-a95b-49e6-9c78-d872d17fd6cb" (UID: "28a8231a-a95b-49e6-9c78-d872d17fd6cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:33.448535 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.448429 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-credential-socket" pod "28a8231a-a95b-49e6-9c78-d872d17fd6cb" (UID: "28a8231a-a95b-49e6-9c78-d872d17fd6cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.448912 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.448852 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.448912 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.448875 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.448912 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.448889 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.448912 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.448904 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.449133 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.448926 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.450628 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.450601 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a8231a-a95b-49e6-9c78-d872d17fd6cb-kube-api-access-5b9gd" pod "28a8231a-a95b-49e6-9c78-d872d17fd6cb" (UID: "28a8231a-a95b-49e6-9c78-d872d17fd6cb"). InnerVolumeSpecName "kube-api-access-5b9gd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:33.450744 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.450674 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-envoy" pod "28a8231a-a95b-49e6-9c78-d872d17fd6cb" (UID: "28a8231a-a95b-49e6-9c78-d872d17fd6cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.450744 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.450691 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-token" pod "28a8231a-a95b-49e6-9c78-d872d17fd6cb" (UID: "28a8231a-a95b-49e6-9c78-d872d17fd6cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:33.450846 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.450798 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-podinfo" pod "28a8231a-a95b-49e6-9c78-d872d17fd6cb" (UID: "28a8231a-a95b-49e6-9c78-d872d17fd6cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:59:33.550604 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.550570 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.550604 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.550598 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.550604 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.550609 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b9gd\" (UniqueName: \"kubernetes.io/projected/28a8231a-a95b-49e6-9c78-d872d17fd6cb-kube-api-access-5b9gd\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.550829 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.550620 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28a8231a-a95b-49e6-9c78-d872d17fd6cb-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.675716 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.675626 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5" event={"ID":"6a0808b2-7798-4f60-9435-063b72bbdb21","Type":"ContainerStarted","Data":"d656819b3d12ec61c757a0300fed5527a29363433d718728766a745f58459813"} Nov 03 20:59:33.677138 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.677109 4493 generic.go:356] "Generic (PLEG): container finished" podID="28a8231a-a95b-49e6-9c78-d872d17fd6cb" containerID="1a8386cb970ec6728645b685d0ea191d47f14f9ded64e64ddaa9d145a0079655" exitCode=0 Nov 03 20:59:33.677285 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.677172 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" Nov 03 20:59:33.677285 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.677182 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" event={"ID":"28a8231a-a95b-49e6-9c78-d872d17fd6cb","Type":"ContainerDied","Data":"1a8386cb970ec6728645b685d0ea191d47f14f9ded64e64ddaa9d145a0079655"} Nov 03 20:59:33.677285 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.677246 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j" event={"ID":"28a8231a-a95b-49e6-9c78-d872d17fd6cb","Type":"ContainerDied","Data":"47c64fd1494c7e773e34f8e0496e32deee0c84da8596b28916d59a5f389d7d9b"} Nov 03 20:59:33.677285 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.677270 4493 scope.go:122] "RemoveContainer" containerID="1a8386cb970ec6728645b685d0ea191d47f14f9ded64e64ddaa9d145a0079655" Nov 03 20:59:33.700899 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.700717 4493 scope.go:122] "RemoveContainer" containerID="1a8386cb970ec6728645b685d0ea191d47f14f9ded64e64ddaa9d145a0079655" Nov 03 20:59:33.703793 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:59:33.701140 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a8386cb970ec6728645b685d0ea191d47f14f9ded64e64ddaa9d145a0079655\": container with ID starting with 1a8386cb970ec6728645b685d0ea191d47f14f9ded64e64ddaa9d145a0079655 not found: ID does not exist" containerID="1a8386cb970ec6728645b685d0ea191d47f14f9ded64e64ddaa9d145a0079655" Nov 03 20:59:33.703793 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.701390 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a8386cb970ec6728645b685d0ea191d47f14f9ded64e64ddaa9d145a0079655"} err="failed to get container status \"1a8386cb970ec6728645b685d0ea191d47f14f9ded64e64ddaa9d145a0079655\": rpc error: code = NotFound desc = could not find container \"1a8386cb970ec6728645b685d0ea191d47f14f9ded64e64ddaa9d145a0079655\": container with ID starting with 1a8386cb970ec6728645b685d0ea191d47f14f9ded64e64ddaa9d145a0079655 not found: ID does not exist" Nov 03 20:59:33.834108 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.834081 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j"] Nov 03 20:59:33.838549 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:33.838520 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4nx7j"] Nov 03 20:59:34.564477 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:34.564433 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28a8231a-a95b-49e6-9c78-d872d17fd6cb" path="/var/lib/kubelet/pods/28a8231a-a95b-49e6-9c78-d872d17fd6cb/volumes" Nov 03 20:59:34.685345 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:34.685288 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5" event={"ID":"6a0808b2-7798-4f60-9435-063b72bbdb21","Type":"ContainerStarted","Data":"46052c58d0c3ed596a21a51897546db2ecf0735a968a75cc99b8d7d9dad390de"} Nov 03 20:59:34.685750 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:34.685393 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5" Nov 03 20:59:34.792893 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:34.792664 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5" podStartSLOduration=2.874637031 podStartE2EDuration="3.792643827s" totalImagesPullingTime="918.006796ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:34.792378425 +0000 UTC m=+1846.350368943" watchObservedRunningTime="2026-11-03 20:59:34.792643827 +0000 UTC m=+1846.350634327" Nov 03 20:59:40.702912 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:40.702881 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5" Nov 03 20:59:49.831202 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:49.831172 4493 scope.go:122] "RemoveContainer" containerID="f0d94138dc8d402716fe206b02410e528179e93b8e70619b6f9ebf40140b914a" Nov 03 20:59:49.841296 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:49.841275 4493 scope.go:122] "RemoveContainer" containerID="09243faedf52d27d133a6cc95cc6f34c38c9ca3652923301215762ec97308cd1" Nov 03 20:59:51.760407 ip-10-0-138-26 kubenswrapper[4493]: E1103 20:59:51.760314 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 20:59:53.247361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:53.246491 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:59:53.439360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:53.438479 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:59:53.633045 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:53.630419 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 20:59:53.637606 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:53.637549 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:59:53.718668 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:53.717521 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 03 20:59:53.752072 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:53.752025 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:59:53.794360 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:53.793461 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:59:53.911459 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:53.896395 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:59:54.120488 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:54.120438 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:59:54.248521 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:54.248418 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:59:54.250081 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:54.250058 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:59:54.439044 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:54.439003 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:59:54.498365 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:54.470584 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:59:54.498365 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:54.448411 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:59:54.675283 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:54.675237 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:59:54.676508 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:54.676460 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 20:59:54.766603 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:54.766536 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:59:54.768146 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:54.768106 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 03 20:59:54.769801 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:54.769772 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:59:54.929618 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:54.929504 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:59:55.083142 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:55.083097 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:59:55.137680 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:55.137645 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:59:55.138063 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:55.137953 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:59:55.312423 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:55.301772 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:59:55.312423 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:55.301822 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 20:59:55.312423 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:55.301862 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:59:55.312423 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:55.301880 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 20:59:55.312423 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:55.301911 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 20:59:55.312423 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:55.301942 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 20:59:55.312423 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:55.301974 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 20:59:55.312423 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:55.302003 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 20:59:55.377531 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:55.377500 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 20:59:56.037778 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.037734 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw"] Nov 03 20:59:56.038717 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.038694 4493 cpu_manager.go:414] "containerMap: removing container" podUID="28a8231a-a95b-49e6-9c78-d872d17fd6cb" containerName="istio-proxy" Nov 03 20:59:56.038788 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.038721 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a8231a-a95b-49e6-9c78-d872d17fd6cb" containerName="istio-proxy" Nov 03 20:59:56.038928 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.038916 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a092ea83-8438-4626-a3a6-0e076627ab11" containerName="istio-proxy" Nov 03 20:59:56.038965 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.038933 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a8231a-a95b-49e6-9c78-d872d17fd6cb" containerName="istio-proxy" Nov 03 20:59:56.038965 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.038948 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f51d926d-5ce9-48da-8444-3042b674b606" containerName="istio-proxy" Nov 03 20:59:56.045033 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.045009 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.114361 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.102862 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-88bc35-openshift-default-dockercfg-t6gcz\"" Nov 03 20:59:56.147353 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.125860 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-data\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.147678 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.147650 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-token\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.147886 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.147866 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-credential-socket\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.147988 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.147908 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw"] Nov 03 20:59:56.151128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.150793 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-workload-certs\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.151128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.150835 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istiod-ca-cert\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.151128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.150925 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ccb4\" (UniqueName: \"kubernetes.io/projected/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-kube-api-access-9ccb4\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.151128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.151061 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-envoy\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.151128 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.151098 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-workload-socket\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.151516 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.151149 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-podinfo\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.268369 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.266522 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-workload-socket\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.268369 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.266620 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-podinfo\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.268369 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.266804 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-data\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.268369 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.266830 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-token\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.268369 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.266905 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-credential-socket\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.268369 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.267025 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-workload-certs\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.268369 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.267055 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istiod-ca-cert\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.268369 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.267115 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ccb4\" (UniqueName: \"kubernetes.io/projected/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-kube-api-access-9ccb4\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.268369 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.267264 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-envoy\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.269023 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.268765 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-credential-socket\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.269023 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.268998 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-workload-socket\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.270810 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.270785 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istiod-ca-cert\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.271251 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.271231 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-workload-certs\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.272251 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.272222 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-data\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.280305 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.280267 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-envoy\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.350492 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.348289 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-podinfo\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.355013 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.354984 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ccb4\" (UniqueName: \"kubernetes.io/projected/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-kube-api-access-9ccb4\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.356375 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.356075 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-token\") pod \"e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.359289 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.359100 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:56.852080 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.852046 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw"] Nov 03 20:59:56.856654 ip-10-0-138-26 kubenswrapper[4493]: W1103 20:59:56.856627 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c88cfbc_20ad_47cb_8138_cabbaf57eb1d.slice/crio-bcc6b99a1be7ab72275815bb59ff648d5822e4ca6eadbb2548c2e640352a66d4 WatchSource:0}: Error finding container bcc6b99a1be7ab72275815bb59ff648d5822e4ca6eadbb2548c2e640352a66d4: Status 404 returned error can't find the container with id bcc6b99a1be7ab72275815bb59ff648d5822e4ca6eadbb2548c2e640352a66d4 Nov 03 20:59:56.859473 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.859441 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:59:56.859557 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.859511 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:59:56.859557 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:56.859543 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 20:59:57.172812 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:57.172721 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" event={"ID":"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d","Type":"ContainerStarted","Data":"398ad7ddae6b1b89e09731f3562f8648ed7f2a0e9e2abbb4158b4db0369da269"} Nov 03 20:59:57.172812 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:57.172783 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" event={"ID":"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d","Type":"ContainerStarted","Data":"bcc6b99a1be7ab72275815bb59ff648d5822e4ca6eadbb2548c2e640352a66d4"} Nov 03 20:59:57.304358 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:57.294808 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" podStartSLOduration=2.294787507 podStartE2EDuration="2.294787507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:57.294125252 +0000 UTC m=+1868.852115776" watchObservedRunningTime="2026-11-03 20:59:57.294787507 +0000 UTC m=+1868.852778007" Nov 03 20:59:57.360505 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:57.360475 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:57.362131 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:57.362097 4493 patch_prober.go:29] interesting pod/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 20:59:57.362258 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:57.362145 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" podUID="8c88cfbc-20ad-47cb-8138-cabbaf57eb1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 20:59:58.365122 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:58.365087 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:59.187515 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:59.187483 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 20:59:59.188705 ip-10-0-138-26 kubenswrapper[4493]: I1103 20:59:59.188680 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 21:00:00.232438 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.228504 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k"] Nov 03 21:00:00.236080 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.236053 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7"] Nov 03 21:00:00.236254 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.236229 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k" Nov 03 21:00:00.241116 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.241088 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl"] Nov 03 21:00:00.241290 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.241195 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7" Nov 03 21:00:00.247416 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.247392 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl" Nov 03 21:00:00.329639 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.329600 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\"" Nov 03 21:00:00.359248 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.359204 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txfd5\" (UniqueName: \"kubernetes.io/projected/c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68-kube-api-access-txfd5\") pod \"maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k\" (UID: \"c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k" Nov 03 21:00:00.359459 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.359270 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00adf07-f852-4913-bddf-b80e4d16ebaa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl\" (UID: \"f00adf07-f852-4913-bddf-b80e4d16ebaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl" Nov 03 21:00:00.359459 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.359304 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd979\" (UniqueName: \"kubernetes.io/projected/f00adf07-f852-4913-bddf-b80e4d16ebaa-kube-api-access-xd979\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl\" (UID: \"f00adf07-f852-4913-bddf-b80e4d16ebaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl" Nov 03 21:00:00.359601 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.359470 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k\" (UID: \"c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k" Nov 03 21:00:00.359601 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.359592 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxw42\" (UniqueName: \"kubernetes.io/projected/79e293ed-531a-47c6-acbc-f9adc4d76642-kube-api-access-rxw42\") pod \"maas-api-key-cleanup-29895660-d6rn7\" (UID: \"79e293ed-531a-47c6-acbc-f9adc4d76642\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7" Nov 03 21:00:00.359726 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.359670 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e293ed-531a-47c6-acbc-f9adc4d76642-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895660-d6rn7\" (UID: \"79e293ed-531a-47c6-acbc-f9adc4d76642\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7" Nov 03 21:00:00.370794 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.370389 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k"] Nov 03 21:00:00.373572 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.373538 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7"] Nov 03 21:00:00.412079 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.412042 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl"] Nov 03 21:00:00.460750 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.460709 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxw42\" (UniqueName: \"kubernetes.io/projected/79e293ed-531a-47c6-acbc-f9adc4d76642-kube-api-access-rxw42\") pod \"maas-api-key-cleanup-29895660-d6rn7\" (UID: \"79e293ed-531a-47c6-acbc-f9adc4d76642\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7" Nov 03 21:00:00.460950 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.460775 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e293ed-531a-47c6-acbc-f9adc4d76642-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895660-d6rn7\" (UID: \"79e293ed-531a-47c6-acbc-f9adc4d76642\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7" Nov 03 21:00:00.460950 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.460897 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txfd5\" (UniqueName: \"kubernetes.io/projected/c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68-kube-api-access-txfd5\") pod \"maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k\" (UID: \"c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k" Nov 03 21:00:00.460950 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.460934 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00adf07-f852-4913-bddf-b80e4d16ebaa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl\" (UID: \"f00adf07-f852-4913-bddf-b80e4d16ebaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl" Nov 03 21:00:00.461143 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.460959 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd979\" (UniqueName: \"kubernetes.io/projected/f00adf07-f852-4913-bddf-b80e4d16ebaa-kube-api-access-xd979\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl\" (UID: \"f00adf07-f852-4913-bddf-b80e4d16ebaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl" Nov 03 21:00:00.461143 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.461083 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k\" (UID: \"c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k" Nov 03 21:00:00.461709 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.461658 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e293ed-531a-47c6-acbc-f9adc4d76642-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895660-d6rn7\" (UID: \"79e293ed-531a-47c6-acbc-f9adc4d76642\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7" Nov 03 21:00:00.461843 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.461767 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00adf07-f852-4913-bddf-b80e4d16ebaa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl\" (UID: \"f00adf07-f852-4913-bddf-b80e4d16ebaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl" Nov 03 21:00:00.461922 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.461857 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k\" (UID: \"c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k" Nov 03 21:00:00.516050 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.515969 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txfd5\" (UniqueName: \"kubernetes.io/projected/c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68-kube-api-access-txfd5\") pod \"maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k\" (UID: \"c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k" Nov 03 21:00:00.516353 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.516319 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxw42\" (UniqueName: \"kubernetes.io/projected/79e293ed-531a-47c6-acbc-f9adc4d76642-kube-api-access-rxw42\") pod \"maas-api-key-cleanup-29895660-d6rn7\" (UID: \"79e293ed-531a-47c6-acbc-f9adc4d76642\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7" Nov 03 21:00:00.516462 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.516350 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd979\" (UniqueName: \"kubernetes.io/projected/f00adf07-f852-4913-bddf-b80e4d16ebaa-kube-api-access-xd979\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl\" (UID: \"f00adf07-f852-4913-bddf-b80e4d16ebaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl" Nov 03 21:00:00.560288 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.560255 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k" Nov 03 21:00:00.569716 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.569686 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7" Nov 03 21:00:00.576770 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:00.576740 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl" Nov 03 21:00:01.018371 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.009144 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7"] Nov 03 21:00:01.032221 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:00:01.024563 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e293ed_531a_47c6_acbc_f9adc4d76642.slice/crio-8aefd6540a7017d7d21568d08555998af5b6ed7554703a3ddb5466972c341873 WatchSource:0}: Error finding container 8aefd6540a7017d7d21568d08555998af5b6ed7554703a3ddb5466972c341873: Status 404 returned error can't find the container with id 8aefd6540a7017d7d21568d08555998af5b6ed7554703a3ddb5466972c341873 Nov 03 21:00:01.061526 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.061481 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl"] Nov 03 21:00:01.066867 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.066805 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k"] Nov 03 21:00:01.070855 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:00:01.070812 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4dcccf5_d8ee_48b2_83ac_1ae5f3223e68.slice/crio-541ff398aa56484c23374781fb5596534a7e83a3eec70601a1d3fed0681b0551 WatchSource:0}: Error finding container 541ff398aa56484c23374781fb5596534a7e83a3eec70601a1d3fed0681b0551: Status 404 returned error can't find the container with id 541ff398aa56484c23374781fb5596534a7e83a3eec70601a1d3fed0681b0551 Nov 03 21:00:01.073700 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:00:01.073639 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf00adf07_f852_4913_bddf_b80e4d16ebaa.slice/crio-157b83af24249c5c4951aa02c21d9568e8a973599812c31595317738f17635da WatchSource:0}: Error finding container 157b83af24249c5c4951aa02c21d9568e8a973599812c31595317738f17635da: Status 404 returned error can't find the container with id 157b83af24249c5c4951aa02c21d9568e8a973599812c31595317738f17635da Nov 03 21:00:01.248830 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.248682 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7" event={"ID":"79e293ed-531a-47c6-acbc-f9adc4d76642","Type":"ContainerStarted","Data":"8aefd6540a7017d7d21568d08555998af5b6ed7554703a3ddb5466972c341873"} Nov 03 21:00:01.251966 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.251926 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl" event={"ID":"f00adf07-f852-4913-bddf-b80e4d16ebaa","Type":"ContainerStarted","Data":"157b83af24249c5c4951aa02c21d9568e8a973599812c31595317738f17635da"} Nov 03 21:00:01.253695 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.253666 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k" event={"ID":"c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68","Type":"ContainerStarted","Data":"541ff398aa56484c23374781fb5596534a7e83a3eec70601a1d3fed0681b0551"} Nov 03 21:00:01.536203 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.536167 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb"] Nov 03 21:00:01.540462 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.540437 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.589071 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.588995 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-istio-data\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.589231 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.589119 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569b73ab-788e-432e-877d-440a81b56965-istiod-ca-cert\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.589231 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.589201 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569b73ab-788e-432e-877d-440a81b56965-istio-podinfo\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.589231 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.589227 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569b73ab-788e-432e-877d-440a81b56965-istio-token\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.589418 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.589309 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-istio-envoy\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.589418 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.589374 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trdk2\" (UniqueName: \"kubernetes.io/projected/569b73ab-788e-432e-877d-440a81b56965-kube-api-access-trdk2\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.589534 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.589460 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-workload-socket\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.589534 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.589506 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-workload-certs\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.589646 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.589590 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-credential-socket\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.589646 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.589622 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-46ca0b-openshift-default-dockercfg-kztl6\"" Nov 03 21:00:01.601305 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.601274 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb"] Nov 03 21:00:01.691195 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.691114 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-workload-socket\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.691195 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.691181 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-workload-certs\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.691489 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.691265 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-credential-socket\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.691489 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.691458 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-istio-data\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.691629 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.691537 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569b73ab-788e-432e-877d-440a81b56965-istiod-ca-cert\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.691629 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.691596 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569b73ab-788e-432e-877d-440a81b56965-istio-podinfo\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.691755 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.691630 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569b73ab-788e-432e-877d-440a81b56965-istio-token\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.691755 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.691701 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-istio-envoy\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.691755 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.691745 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trdk2\" (UniqueName: \"kubernetes.io/projected/569b73ab-788e-432e-877d-440a81b56965-kube-api-access-trdk2\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.691986 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.691957 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-workload-socket\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.692045 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.691996 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-workload-certs\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.692702 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.692673 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569b73ab-788e-432e-877d-440a81b56965-istiod-ca-cert\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.693075 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.693051 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-istio-data\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.693378 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.693308 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-credential-socket\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.694563 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.694541 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-istio-envoy\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.695246 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.695214 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569b73ab-788e-432e-877d-440a81b56965-istio-podinfo\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.746103 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.746066 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569b73ab-788e-432e-877d-440a81b56965-istio-token\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.746275 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.746145 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trdk2\" (UniqueName: \"kubernetes.io/projected/569b73ab-788e-432e-877d-440a81b56965-kube-api-access-trdk2\") pod \"e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:01.860389 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:01.860284 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:02.071522 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:02.071489 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 21:00:02.071737 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:02.071548 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" podUID="c73e8310-a7fd-439b-8f6a-95e99d528e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 21:00:02.262583 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:02.262548 4493 generic.go:356] "Generic (PLEG): container finished" podID="79e293ed-531a-47c6-acbc-f9adc4d76642" containerID="27950facd857237dba7e8a972eb1743d491bcf0853bf0689be15dee01aab31dc" exitCode=60 Nov 03 21:00:02.264511 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:02.262583 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7" event={"ID":"79e293ed-531a-47c6-acbc-f9adc4d76642","Type":"ContainerDied","Data":"27950facd857237dba7e8a972eb1743d491bcf0853bf0689be15dee01aab31dc"} Nov 03 21:00:02.264511 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:02.262990 4493 scope.go:122] "RemoveContainer" containerID="27950facd857237dba7e8a972eb1743d491bcf0853bf0689be15dee01aab31dc" Nov 03 21:00:02.265049 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:02.265021 4493 generic.go:356] "Generic (PLEG): container finished" podID="f00adf07-f852-4913-bddf-b80e4d16ebaa" containerID="83bd2fa295752e3ec213f1262ca8d49348ba4700a069a5074aab1a9f6ba8caf7" exitCode=60 Nov 03 21:00:02.265196 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:02.265131 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl" event={"ID":"f00adf07-f852-4913-bddf-b80e4d16ebaa","Type":"ContainerDied","Data":"83bd2fa295752e3ec213f1262ca8d49348ba4700a069a5074aab1a9f6ba8caf7"} Nov 03 21:00:02.266213 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:02.266192 4493 scope.go:122] "RemoveContainer" containerID="83bd2fa295752e3ec213f1262ca8d49348ba4700a069a5074aab1a9f6ba8caf7" Nov 03 21:00:02.268213 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:02.267689 4493 generic.go:356] "Generic (PLEG): container finished" podID="c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68" containerID="30729ede93c51bddbe7543541fc5274b6a4f0d09d4d6dd1475c2f0ff1b3cf86e" exitCode=60 Nov 03 21:00:02.268213 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:02.267766 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k" event={"ID":"c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68","Type":"ContainerDied","Data":"30729ede93c51bddbe7543541fc5274b6a4f0d09d4d6dd1475c2f0ff1b3cf86e"} Nov 03 21:00:02.268213 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:02.268054 4493 scope.go:122] "RemoveContainer" containerID="30729ede93c51bddbe7543541fc5274b6a4f0d09d4d6dd1475c2f0ff1b3cf86e" Nov 03 21:00:02.270347 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:02.270318 4493 generic.go:356] "Generic (PLEG): container finished" podID="c73e8310-a7fd-439b-8f6a-95e99d528e5d" containerID="9164054bb75d12a8e0d3943ad1a0b1c91cc7ac038ac21cfd3fea388053a265b0" exitCode=0 Nov 03 21:00:02.270471 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:02.270456 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" event={"ID":"c73e8310-a7fd-439b-8f6a-95e99d528e5d","Type":"ContainerDied","Data":"9164054bb75d12a8e0d3943ad1a0b1c91cc7ac038ac21cfd3fea388053a265b0"} Nov 03 21:00:02.270775 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:02.270755 4493 scope.go:122] "RemoveContainer" containerID="9164054bb75d12a8e0d3943ad1a0b1c91cc7ac038ac21cfd3fea388053a265b0" Nov 03 21:00:02.275468 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:02.275169 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:00:02.275468 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:02.275230 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:00:02.275468 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:02.275263 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:00:02.305438 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:02.305410 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb"] Nov 03 21:00:02.327681 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:00:02.327654 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569b73ab_788e_432e_877d_440a81b56965.slice/crio-216e22e2bbb8e001e3e344622b4eb77ea9cf743574ea3f154bb9ef1f64cdc10c WatchSource:0}: Error finding container 216e22e2bbb8e001e3e344622b4eb77ea9cf743574ea3f154bb9ef1f64cdc10c: Status 404 returned error can't find the container with id 216e22e2bbb8e001e3e344622b4eb77ea9cf743574ea3f154bb9ef1f64cdc10c Nov 03 21:00:02.330586 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:02.330552 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:00:02.330684 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:02.330636 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:00:02.330684 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:02.330670 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:00:03.078504 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:03.078463 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 03 21:00:03.078681 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:03.078529 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" podUID="e51960f4-5af5-4d6b-ae00-29b0e54c5ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 03 21:00:03.279183 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:03.279146 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7" event={"ID":"79e293ed-531a-47c6-acbc-f9adc4d76642","Type":"ContainerStarted","Data":"dbdd8e4e4f1165bddefb91d2894c83da42f02bf12592182498c474a27981b291"} Nov 03 21:00:03.281079 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:03.281052 4493 generic.go:356] "Generic (PLEG): container finished" podID="e51960f4-5af5-4d6b-ae00-29b0e54c5ba3" containerID="c566abe6eaaae1012a09bfbac4215eaed8a252ee2323151f7334bcb5c595faaf" exitCode=0 Nov 03 21:00:03.281218 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:03.281120 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" event={"ID":"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3","Type":"ContainerDied","Data":"c566abe6eaaae1012a09bfbac4215eaed8a252ee2323151f7334bcb5c595faaf"} Nov 03 21:00:03.281563 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:03.281546 4493 scope.go:122] "RemoveContainer" containerID="c566abe6eaaae1012a09bfbac4215eaed8a252ee2323151f7334bcb5c595faaf" Nov 03 21:00:03.283294 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:03.283261 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:00:03.283294 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:03.283282 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl" event={"ID":"f00adf07-f852-4913-bddf-b80e4d16ebaa","Type":"ContainerStarted","Data":"7934ad60e2776fd57339736139182de8001e9d3595ede37c952f59908625f05f"} Nov 03 21:00:03.283497 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:03.283364 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:00:03.283497 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:03.283423 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:00:03.286354 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:03.286302 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k" event={"ID":"c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68","Type":"ContainerStarted","Data":"9a555b6979e76cb8d888bb803409235b27936e8ff050d5df6415ae0d0f829cc5"} Nov 03 21:00:03.289009 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:03.288985 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" event={"ID":"c73e8310-a7fd-439b-8f6a-95e99d528e5d","Type":"ContainerStarted","Data":"c7a20fa64416465ab5b5500c1dd2bb9f439ba6035a3d69d0d8fe48ea51fdb4c5"} Nov 03 21:00:03.291756 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:03.291647 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" event={"ID":"569b73ab-788e-432e-877d-440a81b56965","Type":"ContainerStarted","Data":"703b96b868da99fdea69d97e11417864b67982ec20823ee0f42978b32279c9d0"} Nov 03 21:00:03.291756 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:03.291694 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" event={"ID":"569b73ab-788e-432e-877d-440a81b56965","Type":"ContainerStarted","Data":"216e22e2bbb8e001e3e344622b4eb77ea9cf743574ea3f154bb9ef1f64cdc10c"} Nov 03 21:00:03.388847 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:03.388805 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7" podStartSLOduration=3.388792084 podStartE2EDuration="3.388792084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:03.387526994 +0000 UTC m=+1874.945517506" watchObservedRunningTime="2026-11-03 21:00:03.388792084 +0000 UTC m=+1874.946782583" Nov 03 21:00:03.521359 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:03.517432 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 21:00:03.521359 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:03.517824 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 21:00:03.521359 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:03.517875 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" podUID="c73e8310-a7fd-439b-8f6a-95e99d528e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 21:00:03.617047 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:03.616979 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl" podStartSLOduration=3.616955537 podStartE2EDuration="3.616955537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:03.608767244 +0000 UTC m=+1875.166757744" watchObservedRunningTime="2026-11-03 21:00:03.616955537 +0000 UTC m=+1875.174946038" Nov 03 21:00:03.718436 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:03.718316 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k" podStartSLOduration=3.718302402 podStartE2EDuration="3.718302402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:03.714762557 +0000 UTC m=+1875.272753075" watchObservedRunningTime="2026-11-03 21:00:03.718302402 +0000 UTC m=+1875.276292974" Nov 03 21:00:03.832537 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:03.832476 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" podStartSLOduration=2.8324571069999998 podStartE2EDuration="2.832457107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:03.824301284 +0000 UTC m=+1875.382291808" watchObservedRunningTime="2026-11-03 21:00:03.832457107 +0000 UTC m=+1875.390447607" Nov 03 21:00:03.861426 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:03.861394 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:03.867506 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:03.867482 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:04.334679 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:04.334636 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" event={"ID":"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3","Type":"ContainerStarted","Data":"59b490cd40499afb454f31bb49f2f64202e6939b66f858746c7e1d090709fcd8"} Nov 03 21:00:04.340129 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:04.340094 4493 generic.go:356] "Generic (PLEG): container finished" podID="f00adf07-f852-4913-bddf-b80e4d16ebaa" containerID="7934ad60e2776fd57339736139182de8001e9d3595ede37c952f59908625f05f" exitCode=60 Nov 03 21:00:04.340477 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:04.340454 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl" event={"ID":"f00adf07-f852-4913-bddf-b80e4d16ebaa","Type":"ContainerDied","Data":"7934ad60e2776fd57339736139182de8001e9d3595ede37c952f59908625f05f"} Nov 03 21:00:04.340745 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:04.340730 4493 scope.go:122] "RemoveContainer" containerID="83bd2fa295752e3ec213f1262ca8d49348ba4700a069a5074aab1a9f6ba8caf7" Nov 03 21:00:04.341587 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:04.341532 4493 scope.go:122] "RemoveContainer" containerID="7934ad60e2776fd57339736139182de8001e9d3595ede37c952f59908625f05f" Nov 03 21:00:04.342112 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:00:04.342085 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl_odh-ai-gateway-infra(f00adf07-f852-4913-bddf-b80e4d16ebaa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl" podUID="f00adf07-f852-4913-bddf-b80e4d16ebaa" Nov 03 21:00:04.344120 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:04.344095 4493 generic.go:356] "Generic (PLEG): container finished" podID="c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68" containerID="9a555b6979e76cb8d888bb803409235b27936e8ff050d5df6415ae0d0f829cc5" exitCode=60 Nov 03 21:00:04.344287 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:04.344271 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k" event={"ID":"c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68","Type":"ContainerDied","Data":"9a555b6979e76cb8d888bb803409235b27936e8ff050d5df6415ae0d0f829cc5"} Nov 03 21:00:04.344974 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:04.344955 4493 scope.go:122] "RemoveContainer" containerID="9a555b6979e76cb8d888bb803409235b27936e8ff050d5df6415ae0d0f829cc5" Nov 03 21:00:04.345521 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:00:04.345491 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k_odh-ai-gateway-infra(c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k" podUID="c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68" Nov 03 21:00:04.347550 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:04.347524 4493 generic.go:356] "Generic (PLEG): container finished" podID="79e293ed-531a-47c6-acbc-f9adc4d76642" containerID="dbdd8e4e4f1165bddefb91d2894c83da42f02bf12592182498c474a27981b291" exitCode=60 Nov 03 21:00:04.347801 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:04.347782 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7" event={"ID":"79e293ed-531a-47c6-acbc-f9adc4d76642","Type":"ContainerDied","Data":"dbdd8e4e4f1165bddefb91d2894c83da42f02bf12592182498c474a27981b291"} Nov 03 21:00:04.348243 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:04.348233 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:04.348471 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:04.348450 4493 scope.go:122] "RemoveContainer" containerID="dbdd8e4e4f1165bddefb91d2894c83da42f02bf12592182498c474a27981b291" Nov 03 21:00:04.348798 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:00:04.348771 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895660-d6rn7_odh-ai-gateway-infra(79e293ed-531a-47c6-acbc-f9adc4d76642)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7" podUID="79e293ed-531a-47c6-acbc-f9adc4d76642" Nov 03 21:00:04.349145 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:04.349110 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:00:04.355815 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:04.355787 4493 scope.go:122] "RemoveContainer" containerID="30729ede93c51bddbe7543541fc5274b6a4f0d09d4d6dd1475c2f0ff1b3cf86e" Nov 03 21:00:04.371496 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:04.371452 4493 scope.go:122] "RemoveContainer" containerID="27950facd857237dba7e8a972eb1743d491bcf0853bf0689be15dee01aab31dc" Nov 03 21:00:04.526361 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:04.525769 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 21:00:04.526361 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:04.525828 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" podUID="c73e8310-a7fd-439b-8f6a-95e99d528e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 21:00:05.036358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:05.035825 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 21:00:05.036358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:05.035958 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 03 21:00:05.036358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:05.036046 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" podUID="e51960f4-5af5-4d6b-ae00-29b0e54c5ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 03 21:00:05.358883 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:05.358858 4493 scope.go:122] "RemoveContainer" containerID="7934ad60e2776fd57339736139182de8001e9d3595ede37c952f59908625f05f" Nov 03 21:00:05.359375 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:00:05.359285 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl_odh-ai-gateway-infra(f00adf07-f852-4913-bddf-b80e4d16ebaa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl" podUID="f00adf07-f852-4913-bddf-b80e4d16ebaa" Nov 03 21:00:05.360559 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:05.360533 4493 scope.go:122] "RemoveContainer" containerID="9a555b6979e76cb8d888bb803409235b27936e8ff050d5df6415ae0d0f829cc5" Nov 03 21:00:05.360891 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:00:05.360867 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k_odh-ai-gateway-infra(c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k" podUID="c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68" Nov 03 21:00:05.362397 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:05.362380 4493 scope.go:122] "RemoveContainer" containerID="dbdd8e4e4f1165bddefb91d2894c83da42f02bf12592182498c474a27981b291" Nov 03 21:00:05.362664 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:00:05.362638 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895660-d6rn7_odh-ai-gateway-infra(79e293ed-531a-47c6-acbc-f9adc4d76642)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7" podUID="79e293ed-531a-47c6-acbc-f9adc4d76642" Nov 03 21:00:05.518357 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:05.517871 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 21:00:05.518357 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:05.517928 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" podUID="c73e8310-a7fd-439b-8f6a-95e99d528e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 21:00:06.028358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:06.027804 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 03 21:00:06.028358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:06.027875 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" podUID="e51960f4-5af5-4d6b-ae00-29b0e54c5ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 03 21:00:06.531358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:06.526739 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 21:00:06.531358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:06.526807 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" podUID="c73e8310-a7fd-439b-8f6a-95e99d528e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 21:00:07.031363 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:07.030783 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 03 21:00:07.031363 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:07.030849 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" podUID="e51960f4-5af5-4d6b-ae00-29b0e54c5ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 03 21:00:07.524360 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:07.520793 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 21:00:07.524360 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:07.520865 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" podUID="c73e8310-a7fd-439b-8f6a-95e99d528e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 21:00:08.034360 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:08.033857 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 03 21:00:08.034360 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:08.033955 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" podUID="e51960f4-5af5-4d6b-ae00-29b0e54c5ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 03 21:00:08.518360 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:08.517571 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 21:00:08.518360 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:08.517622 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" podUID="c73e8310-a7fd-439b-8f6a-95e99d528e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 21:00:09.030317 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:09.029817 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 03 21:00:09.030317 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:09.029924 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" podUID="e51960f4-5af5-4d6b-ae00-29b0e54c5ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 03 21:00:09.522965 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:09.522934 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 21:00:09.523387 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:09.523364 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 21:00:09.524178 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:09.524155 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 21:00:10.066263 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:10.064166 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 21:00:10.066263 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:10.065539 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 21:00:10.067011 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:10.066904 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 21:00:13.555559 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:13.555527 4493 scope.go:122] "RemoveContainer" containerID="7934ad60e2776fd57339736139182de8001e9d3595ede37c952f59908625f05f" Nov 03 21:00:14.450124 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:14.450088 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl" event={"ID":"f00adf07-f852-4913-bddf-b80e4d16ebaa","Type":"ContainerStarted","Data":"79f9d9de4205cbf3cc4b223ced7fc8129e3da41e3257747d8ff9ae0ac645e402"} Nov 03 21:00:14.555139 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:14.555102 4493 scope.go:122] "RemoveContainer" containerID="dbdd8e4e4f1165bddefb91d2894c83da42f02bf12592182498c474a27981b291" Nov 03 21:00:15.463153 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:15.463110 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7" event={"ID":"79e293ed-531a-47c6-acbc-f9adc4d76642","Type":"ContainerStarted","Data":"f5ea3932df04dc42be73dea1193c7e0b374ea52bb2343fb022901914f2250bc6"} Nov 03 21:00:15.465788 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:15.465764 4493 generic.go:356] "Generic (PLEG): container finished" podID="f00adf07-f852-4913-bddf-b80e4d16ebaa" containerID="79f9d9de4205cbf3cc4b223ced7fc8129e3da41e3257747d8ff9ae0ac645e402" exitCode=60 Nov 03 21:00:15.465942 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:15.465824 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl" event={"ID":"f00adf07-f852-4913-bddf-b80e4d16ebaa","Type":"ContainerDied","Data":"79f9d9de4205cbf3cc4b223ced7fc8129e3da41e3257747d8ff9ae0ac645e402"} Nov 03 21:00:15.465942 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:15.465864 4493 scope.go:122] "RemoveContainer" containerID="7934ad60e2776fd57339736139182de8001e9d3595ede37c952f59908625f05f" Nov 03 21:00:15.466219 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:15.466201 4493 scope.go:122] "RemoveContainer" containerID="79f9d9de4205cbf3cc4b223ced7fc8129e3da41e3257747d8ff9ae0ac645e402" Nov 03 21:00:15.466608 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:00:15.466579 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl_odh-ai-gateway-infra(f00adf07-f852-4913-bddf-b80e4d16ebaa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl" podUID="f00adf07-f852-4913-bddf-b80e4d16ebaa" Nov 03 21:00:15.555370 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:15.555315 4493 scope.go:122] "RemoveContainer" containerID="9a555b6979e76cb8d888bb803409235b27936e8ff050d5df6415ae0d0f829cc5" Nov 03 21:00:15.581731 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:15.581707 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl"] Nov 03 21:00:16.474601 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:16.474568 4493 generic.go:356] "Generic (PLEG): container finished" podID="79e293ed-531a-47c6-acbc-f9adc4d76642" containerID="f5ea3932df04dc42be73dea1193c7e0b374ea52bb2343fb022901914f2250bc6" exitCode=60 Nov 03 21:00:16.475049 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:16.474641 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7" event={"ID":"79e293ed-531a-47c6-acbc-f9adc4d76642","Type":"ContainerDied","Data":"f5ea3932df04dc42be73dea1193c7e0b374ea52bb2343fb022901914f2250bc6"} Nov 03 21:00:16.475049 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:16.474701 4493 scope.go:122] "RemoveContainer" containerID="dbdd8e4e4f1165bddefb91d2894c83da42f02bf12592182498c474a27981b291" Nov 03 21:00:16.475165 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:16.475097 4493 scope.go:122] "RemoveContainer" containerID="f5ea3932df04dc42be73dea1193c7e0b374ea52bb2343fb022901914f2250bc6" Nov 03 21:00:16.475542 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:00:16.475515 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895660-d6rn7_odh-ai-gateway-infra(79e293ed-531a-47c6-acbc-f9adc4d76642)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7" podUID="79e293ed-531a-47c6-acbc-f9adc4d76642" Nov 03 21:00:16.478448 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:16.478428 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k" event={"ID":"c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68","Type":"ContainerStarted","Data":"babf9e9472a6d36831990cd79804e8c1c09501f98ee83e93951fb064f7efa503"} Nov 03 21:00:16.582643 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:16.582616 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7"] Nov 03 21:00:16.703018 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:16.702989 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl" Nov 03 21:00:16.796372 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:16.796346 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f00adf07-f852-4913-bddf-b80e4d16ebaa-kube-api-access-xd979\" (UniqueName: \"kubernetes.io/projected/f00adf07-f852-4913-bddf-b80e4d16ebaa-kube-api-access-xd979\") pod \"f00adf07-f852-4913-bddf-b80e4d16ebaa\" (UID: \"f00adf07-f852-4913-bddf-b80e4d16ebaa\") " Nov 03 21:00:16.796542 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:16.796490 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f00adf07-f852-4913-bddf-b80e4d16ebaa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00adf07-f852-4913-bddf-b80e4d16ebaa-maas-api-service-ca\") pod \"f00adf07-f852-4913-bddf-b80e4d16ebaa\" (UID: \"f00adf07-f852-4913-bddf-b80e4d16ebaa\") " Nov 03 21:00:16.796876 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:16.796844 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f00adf07-f852-4913-bddf-b80e4d16ebaa-maas-api-service-ca" pod "f00adf07-f852-4913-bddf-b80e4d16ebaa" (UID: "f00adf07-f852-4913-bddf-b80e4d16ebaa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:16.798625 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:16.798600 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f00adf07-f852-4913-bddf-b80e4d16ebaa-kube-api-access-xd979" pod "f00adf07-f852-4913-bddf-b80e4d16ebaa" (UID: "f00adf07-f852-4913-bddf-b80e4d16ebaa"). InnerVolumeSpecName "kube-api-access-xd979". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:16.897965 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:16.897931 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd979\" (UniqueName: \"kubernetes.io/projected/f00adf07-f852-4913-bddf-b80e4d16ebaa-kube-api-access-xd979\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:16.897965 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:16.897966 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00adf07-f852-4913-bddf-b80e4d16ebaa-maas-api-service-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:17.512394 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.509909 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl" event={"ID":"f00adf07-f852-4913-bddf-b80e4d16ebaa","Type":"ContainerDied","Data":"157b83af24249c5c4951aa02c21d9568e8a973599812c31595317738f17635da"} Nov 03 21:00:17.512920 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.512891 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl" Nov 03 21:00:17.519495 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.519466 4493 scope.go:122] "RemoveContainer" containerID="79f9d9de4205cbf3cc4b223ced7fc8129e3da41e3257747d8ff9ae0ac645e402" Nov 03 21:00:17.522781 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.522731 4493 generic.go:356] "Generic (PLEG): container finished" podID="c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68" containerID="babf9e9472a6d36831990cd79804e8c1c09501f98ee83e93951fb064f7efa503" exitCode=60 Nov 03 21:00:17.523228 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.522985 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k" event={"ID":"c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68","Type":"ContainerDied","Data":"babf9e9472a6d36831990cd79804e8c1c09501f98ee83e93951fb064f7efa503"} Nov 03 21:00:17.524061 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.523547 4493 scope.go:122] "RemoveContainer" containerID="babf9e9472a6d36831990cd79804e8c1c09501f98ee83e93951fb064f7efa503" Nov 03 21:00:17.524061 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:00:17.523969 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k_odh-ai-gateway-infra(c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k" podUID="c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68" Nov 03 21:00:17.532735 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.532719 4493 scope.go:122] "RemoveContainer" containerID="9a555b6979e76cb8d888bb803409235b27936e8ff050d5df6415ae0d0f829cc5" Nov 03 21:00:17.607275 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.606902 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv"] Nov 03 21:00:17.607948 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.607890 4493 cpu_manager.go:414] "containerMap: removing container" podUID="f00adf07-f852-4913-bddf-b80e4d16ebaa" containerName="cleanup" Nov 03 21:00:17.607948 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.607914 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f00adf07-f852-4913-bddf-b80e4d16ebaa" containerName="cleanup" Nov 03 21:00:17.607948 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.607939 4493 cpu_manager.go:414] "containerMap: removing container" podUID="f00adf07-f852-4913-bddf-b80e4d16ebaa" containerName="cleanup" Nov 03 21:00:17.607948 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.607944 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f00adf07-f852-4913-bddf-b80e4d16ebaa" containerName="cleanup" Nov 03 21:00:17.608362 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.608065 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f00adf07-f852-4913-bddf-b80e4d16ebaa" containerName="cleanup" Nov 03 21:00:17.608362 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.608080 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f00adf07-f852-4913-bddf-b80e4d16ebaa" containerName="cleanup" Nov 03 21:00:17.618849 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.618824 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv" Nov 03 21:00:17.672896 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.671961 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv"] Nov 03 21:00:17.672896 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.672486 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-88bc35-dockercfg-66jgn\"" Nov 03 21:00:17.672896 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.672586 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-88bc35\"" Nov 03 21:00:17.748901 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.748863 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qjzs\" (UniqueName: \"kubernetes.io/projected/43ec7143-f92e-406d-82e4-9c71bb5ff7ac-kube-api-access-9qjzs\") pod \"payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv\" (UID: \"43ec7143-f92e-406d-82e4-9c71bb5ff7ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv" Nov 03 21:00:17.759076 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.759025 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ec7143-f92e-406d-82e4-9c71bb5ff7ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv\" (UID: \"43ec7143-f92e-406d-82e4-9c71bb5ff7ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv" Nov 03 21:00:17.784768 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.784688 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k"] Nov 03 21:00:17.849360 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.825158 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7" Nov 03 21:00:17.860457 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.860421 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79e293ed-531a-47c6-acbc-f9adc4d76642-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e293ed-531a-47c6-acbc-f9adc4d76642-maas-api-service-ca\") pod \"79e293ed-531a-47c6-acbc-f9adc4d76642\" (UID: \"79e293ed-531a-47c6-acbc-f9adc4d76642\") " Nov 03 21:00:17.860716 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.860699 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e293ed-531a-47c6-acbc-f9adc4d76642-kube-api-access-rxw42\" (UniqueName: \"kubernetes.io/projected/79e293ed-531a-47c6-acbc-f9adc4d76642-kube-api-access-rxw42\") pod \"79e293ed-531a-47c6-acbc-f9adc4d76642\" (UID: \"79e293ed-531a-47c6-acbc-f9adc4d76642\") " Nov 03 21:00:17.861212 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.861183 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qjzs\" (UniqueName: \"kubernetes.io/projected/43ec7143-f92e-406d-82e4-9c71bb5ff7ac-kube-api-access-9qjzs\") pod \"payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv\" (UID: \"43ec7143-f92e-406d-82e4-9c71bb5ff7ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv" Nov 03 21:00:17.861482 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.861462 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ec7143-f92e-406d-82e4-9c71bb5ff7ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv\" (UID: \"43ec7143-f92e-406d-82e4-9c71bb5ff7ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv" Nov 03 21:00:17.868555 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.868521 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ec7143-f92e-406d-82e4-9c71bb5ff7ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv\" (UID: \"43ec7143-f92e-406d-82e4-9c71bb5ff7ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv" Nov 03 21:00:17.869086 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.869054 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e293ed-531a-47c6-acbc-f9adc4d76642-maas-api-service-ca" pod "79e293ed-531a-47c6-acbc-f9adc4d76642" (UID: "79e293ed-531a-47c6-acbc-f9adc4d76642"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:17.872913 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.872881 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e293ed-531a-47c6-acbc-f9adc4d76642-kube-api-access-rxw42" pod "79e293ed-531a-47c6-acbc-f9adc4d76642" (UID: "79e293ed-531a-47c6-acbc-f9adc4d76642"). InnerVolumeSpecName "kube-api-access-rxw42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:17.899522 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.899492 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8"] Nov 03 21:00:17.900779 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.900754 4493 cpu_manager.go:414] "containerMap: removing container" podUID="79e293ed-531a-47c6-acbc-f9adc4d76642" containerName="cleanup" Nov 03 21:00:17.900963 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.900950 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e293ed-531a-47c6-acbc-f9adc4d76642" containerName="cleanup" Nov 03 21:00:17.901078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.901066 4493 cpu_manager.go:414] "containerMap: removing container" podUID="79e293ed-531a-47c6-acbc-f9adc4d76642" containerName="cleanup" Nov 03 21:00:17.901177 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.901164 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e293ed-531a-47c6-acbc-f9adc4d76642" containerName="cleanup" Nov 03 21:00:17.901283 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.901272 4493 cpu_manager.go:414] "containerMap: removing container" podUID="79e293ed-531a-47c6-acbc-f9adc4d76642" containerName="cleanup" Nov 03 21:00:17.901375 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.901365 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e293ed-531a-47c6-acbc-f9adc4d76642" containerName="cleanup" Nov 03 21:00:17.901491 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.901481 4493 cpu_manager.go:414] "containerMap: removing container" podUID="f00adf07-f852-4913-bddf-b80e4d16ebaa" containerName="cleanup" Nov 03 21:00:17.901565 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.901555 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f00adf07-f852-4913-bddf-b80e4d16ebaa" containerName="cleanup" Nov 03 21:00:17.901753 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.901744 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e293ed-531a-47c6-acbc-f9adc4d76642" containerName="cleanup" Nov 03 21:00:17.901817 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.901810 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f00adf07-f852-4913-bddf-b80e4d16ebaa" containerName="cleanup" Nov 03 21:00:17.901874 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.901867 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e293ed-531a-47c6-acbc-f9adc4d76642" containerName="cleanup" Nov 03 21:00:17.901917 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.901910 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e293ed-531a-47c6-acbc-f9adc4d76642" containerName="cleanup" Nov 03 21:00:17.906018 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.906000 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8" Nov 03 21:00:17.912494 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.912464 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8"] Nov 03 21:00:17.928407 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.927713 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qjzs\" (UniqueName: \"kubernetes.io/projected/43ec7143-f92e-406d-82e4-9c71bb5ff7ac-kube-api-access-9qjzs\") pod \"payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv\" (UID: \"43ec7143-f92e-406d-82e4-9c71bb5ff7ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv" Nov 03 21:00:17.938911 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.938885 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv" Nov 03 21:00:17.954700 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.954673 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl"] Nov 03 21:00:17.958467 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.958444 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895660-r24sl"] Nov 03 21:00:17.965581 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.965554 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4d5ceb0-75f0-420a-8e62-26eadb0ee960-plugins-config-volume\") pod \"payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8\" (UID: \"d4d5ceb0-75f0-420a-8e62-26eadb0ee960\") " pod="openshift-ingress/payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8" Nov 03 21:00:17.965690 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.965600 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxtk2\" (UniqueName: \"kubernetes.io/projected/d4d5ceb0-75f0-420a-8e62-26eadb0ee960-kube-api-access-pxtk2\") pod \"payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8\" (UID: \"d4d5ceb0-75f0-420a-8e62-26eadb0ee960\") " pod="openshift-ingress/payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8" Nov 03 21:00:17.965820 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.965806 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e293ed-531a-47c6-acbc-f9adc4d76642-maas-api-service-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:17.965856 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:17.965824 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxw42\" (UniqueName: \"kubernetes.io/projected/79e293ed-531a-47c6-acbc-f9adc4d76642-kube-api-access-rxw42\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:18.067560 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:18.067531 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4d5ceb0-75f0-420a-8e62-26eadb0ee960-plugins-config-volume\") pod \"payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8\" (UID: \"d4d5ceb0-75f0-420a-8e62-26eadb0ee960\") " pod="openshift-ingress/payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8" Nov 03 21:00:18.067747 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:18.067591 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxtk2\" (UniqueName: \"kubernetes.io/projected/d4d5ceb0-75f0-420a-8e62-26eadb0ee960-kube-api-access-pxtk2\") pod \"payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8\" (UID: \"d4d5ceb0-75f0-420a-8e62-26eadb0ee960\") " pod="openshift-ingress/payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8" Nov 03 21:00:18.068622 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:18.068589 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4d5ceb0-75f0-420a-8e62-26eadb0ee960-plugins-config-volume\") pod \"payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8\" (UID: \"d4d5ceb0-75f0-420a-8e62-26eadb0ee960\") " pod="openshift-ingress/payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8" Nov 03 21:00:18.121714 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:18.121685 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxtk2\" (UniqueName: \"kubernetes.io/projected/d4d5ceb0-75f0-420a-8e62-26eadb0ee960-kube-api-access-pxtk2\") pod \"payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8\" (UID: \"d4d5ceb0-75f0-420a-8e62-26eadb0ee960\") " pod="openshift-ingress/payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8" Nov 03 21:00:18.223033 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:18.223002 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8" Nov 03 21:00:18.322782 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:18.322717 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv"] Nov 03 21:00:18.327937 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:00:18.327902 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ec7143_f92e_406d_82e4_9c71bb5ff7ac.slice/crio-4da11d8554a9f857c98b2916f9dd4b7e45cab824145407b33904201785af589b WatchSource:0}: Error finding container 4da11d8554a9f857c98b2916f9dd4b7e45cab824145407b33904201785af589b: Status 404 returned error can't find the container with id 4da11d8554a9f857c98b2916f9dd4b7e45cab824145407b33904201785af589b Nov 03 21:00:18.537921 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:18.537886 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv" event={"ID":"43ec7143-f92e-406d-82e4-9c71bb5ff7ac","Type":"ContainerStarted","Data":"bcd76ddcddb59773709fc7a56a799bd2c74339f191b8f987142727f1ead01f2c"} Nov 03 21:00:18.538393 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:18.537949 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv" Nov 03 21:00:18.538393 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:18.537968 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv" event={"ID":"43ec7143-f92e-406d-82e4-9c71bb5ff7ac","Type":"ContainerStarted","Data":"4da11d8554a9f857c98b2916f9dd4b7e45cab824145407b33904201785af589b"} Nov 03 21:00:18.539309 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:18.539288 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7" Nov 03 21:00:18.539454 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:18.539283 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7" event={"ID":"79e293ed-531a-47c6-acbc-f9adc4d76642","Type":"ContainerDied","Data":"8aefd6540a7017d7d21568d08555998af5b6ed7554703a3ddb5466972c341873"} Nov 03 21:00:18.539514 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:18.539452 4493 scope.go:122] "RemoveContainer" containerID="f5ea3932df04dc42be73dea1193c7e0b374ea52bb2343fb022901914f2250bc6" Nov 03 21:00:18.563554 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:18.563521 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f00adf07-f852-4913-bddf-b80e4d16ebaa" path="/var/lib/kubelet/pods/f00adf07-f852-4913-bddf-b80e4d16ebaa/volumes" Nov 03 21:00:18.603270 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:18.603235 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8"] Nov 03 21:00:18.617189 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:00:18.617154 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d5ceb0_75f0_420a_8e62_26eadb0ee960.slice/crio-f47080c01530f52de930645cab3210104dd76b2901ca7fa7b1feb6a51ba9ad36 WatchSource:0}: Error finding container f47080c01530f52de930645cab3210104dd76b2901ca7fa7b1feb6a51ba9ad36: Status 404 returned error can't find the container with id f47080c01530f52de930645cab3210104dd76b2901ca7fa7b1feb6a51ba9ad36 Nov 03 21:00:18.677786 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:18.677761 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k" Nov 03 21:00:18.755030 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:18.755000 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7"] Nov 03 21:00:18.769800 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:18.769777 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-d6rn7"] Nov 03 21:00:18.777470 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:18.777448 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68-maas-api-service-ca\") pod \"c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68\" (UID: \"c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68\") " Nov 03 21:00:18.777580 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:18.777549 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68-kube-api-access-txfd5\" (UniqueName: \"kubernetes.io/projected/c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68-kube-api-access-txfd5\") pod \"c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68\" (UID: \"c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68\") " Nov 03 21:00:18.777805 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:18.777784 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68-maas-api-service-ca" pod "c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68" (UID: "c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:18.777946 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:18.777934 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68-maas-api-service-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:18.779680 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:18.779661 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68-kube-api-access-txfd5" pod "c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68" (UID: "c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68"). InnerVolumeSpecName "kube-api-access-txfd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:18.865941 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:18.865834 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv" podStartSLOduration=1.865818273 podStartE2EDuration="1.865818273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:18.861875066 +0000 UTC m=+1890.419865588" watchObservedRunningTime="2026-11-03 21:00:18.865818273 +0000 UTC m=+1890.423808774" Nov 03 21:00:18.879761 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:18.879736 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txfd5\" (UniqueName: \"kubernetes.io/projected/c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68-kube-api-access-txfd5\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:19.547841 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:19.547804 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8" event={"ID":"d4d5ceb0-75f0-420a-8e62-26eadb0ee960","Type":"ContainerStarted","Data":"fc734675b5bba411266e830bb569aa1f484cfcea0209281e80dfc551344ce3a7"} Nov 03 21:00:19.548392 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:19.547889 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8" event={"ID":"d4d5ceb0-75f0-420a-8e62-26eadb0ee960","Type":"ContainerStarted","Data":"f47080c01530f52de930645cab3210104dd76b2901ca7fa7b1feb6a51ba9ad36"} Nov 03 21:00:19.548392 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:19.548013 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8" Nov 03 21:00:19.549651 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:19.549631 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k" event={"ID":"c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68","Type":"ContainerDied","Data":"541ff398aa56484c23374781fb5596534a7e83a3eec70601a1d3fed0681b0551"} Nov 03 21:00:19.549651 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:19.549646 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k" Nov 03 21:00:19.549858 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:19.549671 4493 scope.go:122] "RemoveContainer" containerID="babf9e9472a6d36831990cd79804e8c1c09501f98ee83e93951fb064f7efa503" Nov 03 21:00:19.657533 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:19.657485 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8" podStartSLOduration=2.657470163 podStartE2EDuration="2.657470163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:19.65446295 +0000 UTC m=+1891.212453469" watchObservedRunningTime="2026-11-03 21:00:19.657470163 +0000 UTC m=+1891.215460662" Nov 03 21:00:19.766825 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:19.766398 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k"] Nov 03 21:00:19.777122 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:19.770320 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895660-s5g6k"] Nov 03 21:00:20.562960 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:20.562915 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79e293ed-531a-47c6-acbc-f9adc4d76642" path="/var/lib/kubelet/pods/79e293ed-531a-47c6-acbc-f9adc4d76642/volumes" Nov 03 21:00:20.563609 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:20.563492 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68" path="/var/lib/kubelet/pods/c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68/volumes" Nov 03 21:00:21.746194 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:00:21.746161 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:00:29.553855 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:29.553823 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv" Nov 03 21:00:30.584116 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:30.584031 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8" Nov 03 21:00:38.662630 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.662591 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm"] Nov 03 21:00:38.663298 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.663279 4493 cpu_manager.go:414] "containerMap: removing container" podUID="c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68" containerName="cleanup" Nov 03 21:00:38.663374 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.663300 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68" containerName="cleanup" Nov 03 21:00:38.663417 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.663375 4493 cpu_manager.go:414] "containerMap: removing container" podUID="c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68" containerName="cleanup" Nov 03 21:00:38.663417 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.663382 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68" containerName="cleanup" Nov 03 21:00:38.663490 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.663479 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68" containerName="cleanup" Nov 03 21:00:38.663523 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.663490 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68" containerName="cleanup" Nov 03 21:00:38.668796 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.668776 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.718488 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.718452 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-80cbc0-openshift-default-dockercfg-2qt6j\"" Nov 03 21:00:38.769197 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.769166 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm"] Nov 03 21:00:38.792055 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.792013 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c81f07e-ff42-4534-856a-01f0069a8cb2-istiod-ca-cert\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.792270 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.792177 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-token\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.792358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.792316 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-credential-socket\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.792460 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.792432 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9lcq\" (UniqueName: \"kubernetes.io/projected/2c81f07e-ff42-4534-856a-01f0069a8cb2-kube-api-access-p9lcq\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.792525 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.792466 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-data\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.792525 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.792497 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-podinfo\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.792628 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.792577 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-workload-socket\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.792715 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.792695 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-envoy\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.792773 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.792736 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-workload-certs\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.893926 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.893884 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-credential-socket\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.895712 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.894717 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9lcq\" (UniqueName: \"kubernetes.io/projected/2c81f07e-ff42-4534-856a-01f0069a8cb2-kube-api-access-p9lcq\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.895712 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.894754 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-data\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.895712 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.894783 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-podinfo\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.895712 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.894820 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-workload-socket\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.895712 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.894928 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-envoy\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.895712 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.894963 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-workload-certs\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.895712 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.895118 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c81f07e-ff42-4534-856a-01f0069a8cb2-istiod-ca-cert\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.895712 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.895214 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-token\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.896643 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.894612 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-credential-socket\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.897230 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.897208 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-data\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.898607 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.898587 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-workload-certs\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.899043 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.898996 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-workload-socket\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.899676 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.899656 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c81f07e-ff42-4534-856a-01f0069a8cb2-istiod-ca-cert\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.902872 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.901231 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-envoy\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.906384 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.906321 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-podinfo\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.950915 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.950838 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-token\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.951279 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.951255 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9lcq\" (UniqueName: \"kubernetes.io/projected/2c81f07e-ff42-4534-856a-01f0069a8cb2-kube-api-access-p9lcq\") pod \"e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:38.983507 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:38.983473 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:39.422658 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:39.422621 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm"] Nov 03 21:00:39.427021 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:00:39.426977 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c81f07e_ff42_4534_856a_01f0069a8cb2.slice/crio-66b07a0a2a4b1185052ed70568c5222d9263b3c8ea06af18c78ddd51c0e8262a WatchSource:0}: Error finding container 66b07a0a2a4b1185052ed70568c5222d9263b3c8ea06af18c78ddd51c0e8262a: Status 404 returned error can't find the container with id 66b07a0a2a4b1185052ed70568c5222d9263b3c8ea06af18c78ddd51c0e8262a Nov 03 21:00:39.429956 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:39.429922 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:00:39.430043 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:39.429999 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:00:39.430043 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:39.430032 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:00:39.724028 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:39.723926 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" event={"ID":"2c81f07e-ff42-4534-856a-01f0069a8cb2","Type":"ContainerStarted","Data":"61ea581538c24a83c09a29ac1953b83ee2007762b6e3f0cd1167d00aa0ed7b59"} Nov 03 21:00:39.724028 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:39.723985 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" event={"ID":"2c81f07e-ff42-4534-856a-01f0069a8cb2","Type":"ContainerStarted","Data":"66b07a0a2a4b1185052ed70568c5222d9263b3c8ea06af18c78ddd51c0e8262a"} Nov 03 21:00:39.838146 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:39.838078 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" podStartSLOduration=1.837765995 podStartE2EDuration="1.837765995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:39.832944607 +0000 UTC m=+1911.390935112" watchObservedRunningTime="2026-11-03 21:00:39.837765995 +0000 UTC m=+1911.395756495" Nov 03 21:00:39.984719 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:39.984624 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:40.990721 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:40.990690 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:41.740934 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:41.740894 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:41.742140 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:41.742118 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:00:48.864032 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:48.863998 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd"] Nov 03 21:00:48.864656 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:48.864639 4493 cpu_manager.go:414] "containerMap: removing container" podUID="c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68" containerName="cleanup" Nov 03 21:00:48.864705 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:48.864657 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68" containerName="cleanup" Nov 03 21:00:48.864791 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:48.864780 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4dcccf5-d8ee-48b2-83ac-1ae5f3223e68" containerName="cleanup" Nov 03 21:00:48.868395 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:48.868370 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:48.922767 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:48.922734 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-c63dc3-openshift-default-dockercfg-rwscv\"" Nov 03 21:00:49.001004 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.000974 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd"] Nov 03 21:00:49.010788 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.010693 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-credential-socket\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.013014 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.012985 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-istio-data\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.013363 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.013307 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf7jx\" (UniqueName: \"kubernetes.io/projected/46f74fa9-7638-44fa-b303-813d37808176-kube-api-access-qf7jx\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.013576 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.013557 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f74fa9-7638-44fa-b303-813d37808176-istio-podinfo\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.013735 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.013721 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f74fa9-7638-44fa-b303-813d37808176-istio-token\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.013926 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.013910 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-workload-certs\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.014060 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.014046 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-istio-envoy\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.014228 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.014212 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-workload-socket\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.014381 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.014366 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f74fa9-7638-44fa-b303-813d37808176-istiod-ca-cert\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.117096 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.117022 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-credential-socket\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.117096 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.117084 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-istio-data\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.117327 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.117131 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf7jx\" (UniqueName: \"kubernetes.io/projected/46f74fa9-7638-44fa-b303-813d37808176-kube-api-access-qf7jx\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.117327 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.117166 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f74fa9-7638-44fa-b303-813d37808176-istio-podinfo\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.117327 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.117200 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f74fa9-7638-44fa-b303-813d37808176-istio-token\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.117327 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.117258 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-workload-certs\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.117327 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.117281 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-istio-envoy\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.117620 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.117350 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-workload-socket\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.117620 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.117376 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f74fa9-7638-44fa-b303-813d37808176-istiod-ca-cert\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.117620 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.117568 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-istio-data\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.117784 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.117704 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-workload-certs\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.117784 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.117747 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-workload-socket\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.117878 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.117831 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-credential-socket\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.118036 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.118019 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f74fa9-7638-44fa-b303-813d37808176-istiod-ca-cert\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.119970 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.119946 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-istio-envoy\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.120104 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.119987 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f74fa9-7638-44fa-b303-813d37808176-istio-podinfo\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.174110 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.174074 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f74fa9-7638-44fa-b303-813d37808176-istio-token\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.174267 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.174126 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf7jx\" (UniqueName: \"kubernetes.io/projected/46f74fa9-7638-44fa-b303-813d37808176-kube-api-access-qf7jx\") pod \"e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.208595 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.208557 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:49.654279 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.654242 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd"] Nov 03 21:00:49.659277 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:00:49.659253 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f74fa9_7638_44fa_b303_813d37808176.slice/crio-f91cb11a92c521d09787163112b6f0756610281b687731ac4d2bf62eb45f4b57 WatchSource:0}: Error finding container f91cb11a92c521d09787163112b6f0756610281b687731ac4d2bf62eb45f4b57: Status 404 returned error can't find the container with id f91cb11a92c521d09787163112b6f0756610281b687731ac4d2bf62eb45f4b57 Nov 03 21:00:49.662049 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.662011 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:00:49.662138 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.662082 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:00:49.662138 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.662117 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:00:49.799082 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.799043 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" event={"ID":"46f74fa9-7638-44fa-b303-813d37808176","Type":"ContainerStarted","Data":"389ff94293cd7d3a448630714c20f13a9699dd5cf4ac30bc030c18b5aad79b94"} Nov 03 21:00:49.799558 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.799104 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" event={"ID":"46f74fa9-7638-44fa-b303-813d37808176","Type":"ContainerStarted","Data":"f91cb11a92c521d09787163112b6f0756610281b687731ac4d2bf62eb45f4b57"} Nov 03 21:00:49.914592 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:49.914536 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" podStartSLOduration=1.914516639 podStartE2EDuration="1.914516639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:49.913688043 +0000 UTC m=+1921.471678570" watchObservedRunningTime="2026-11-03 21:00:49.914516639 +0000 UTC m=+1921.472507141" Nov 03 21:00:50.209445 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:50.209362 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:50.216314 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:50.216290 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:50.818587 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:50.818552 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:50.819614 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:50.819586 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:00:51.745914 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:00:51.745872 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:00:58.073534 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:58.073491 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt"] Nov 03 21:00:58.079864 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:58.079838 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt" Nov 03 21:00:58.132235 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:58.132204 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt"] Nov 03 21:00:58.182362 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:58.179607 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-c63dc3\"" Nov 03 21:00:58.182362 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:58.179752 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-c63dc3-dockercfg-5k98d\"" Nov 03 21:00:58.234070 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:58.233960 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c60698d-6705-4cfc-8a08-4246536566b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt\" (UID: \"0c60698d-6705-4cfc-8a08-4246536566b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt" Nov 03 21:00:58.234443 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:58.234416 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf9dd\" (UniqueName: \"kubernetes.io/projected/0c60698d-6705-4cfc-8a08-4246536566b5-kube-api-access-mf9dd\") pod \"payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt\" (UID: \"0c60698d-6705-4cfc-8a08-4246536566b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt" Nov 03 21:00:58.250310 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:58.250230 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg"] Nov 03 21:00:58.259851 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:58.259821 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg" Nov 03 21:00:58.283788 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:58.283732 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg"] Nov 03 21:00:58.347208 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:58.347129 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a3fc99-772f-4628-b9bc-7754ae9e4b1f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg\" (UID: \"74a3fc99-772f-4628-b9bc-7754ae9e4b1f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg" Nov 03 21:00:58.347557 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:58.347532 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkf9h\" (UniqueName: \"kubernetes.io/projected/74a3fc99-772f-4628-b9bc-7754ae9e4b1f-kube-api-access-rkf9h\") pod \"payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg\" (UID: \"74a3fc99-772f-4628-b9bc-7754ae9e4b1f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg" Nov 03 21:00:58.347680 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:58.347652 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c60698d-6705-4cfc-8a08-4246536566b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt\" (UID: \"0c60698d-6705-4cfc-8a08-4246536566b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt" Nov 03 21:00:58.347772 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:58.347755 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf9dd\" (UniqueName: \"kubernetes.io/projected/0c60698d-6705-4cfc-8a08-4246536566b5-kube-api-access-mf9dd\") pod \"payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt\" (UID: \"0c60698d-6705-4cfc-8a08-4246536566b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt" Nov 03 21:00:58.348684 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:58.348614 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c60698d-6705-4cfc-8a08-4246536566b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt\" (UID: \"0c60698d-6705-4cfc-8a08-4246536566b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt" Nov 03 21:00:58.402410 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:58.402379 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf9dd\" (UniqueName: \"kubernetes.io/projected/0c60698d-6705-4cfc-8a08-4246536566b5-kube-api-access-mf9dd\") pod \"payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt\" (UID: \"0c60698d-6705-4cfc-8a08-4246536566b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt" Nov 03 21:00:58.449696 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:58.449654 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a3fc99-772f-4628-b9bc-7754ae9e4b1f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg\" (UID: \"74a3fc99-772f-4628-b9bc-7754ae9e4b1f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg" Nov 03 21:00:58.449843 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:58.449749 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkf9h\" (UniqueName: \"kubernetes.io/projected/74a3fc99-772f-4628-b9bc-7754ae9e4b1f-kube-api-access-rkf9h\") pod \"payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg\" (UID: \"74a3fc99-772f-4628-b9bc-7754ae9e4b1f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg" Nov 03 21:00:58.450390 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:58.450371 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a3fc99-772f-4628-b9bc-7754ae9e4b1f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg\" (UID: \"74a3fc99-772f-4628-b9bc-7754ae9e4b1f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg" Nov 03 21:00:58.504917 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:58.504881 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkf9h\" (UniqueName: \"kubernetes.io/projected/74a3fc99-772f-4628-b9bc-7754ae9e4b1f-kube-api-access-rkf9h\") pod \"payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg\" (UID: \"74a3fc99-772f-4628-b9bc-7754ae9e4b1f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg" Nov 03 21:00:58.579527 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:58.579500 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg" Nov 03 21:00:58.694910 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:58.694874 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt" Nov 03 21:00:58.954888 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:58.954517 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg"] Nov 03 21:00:58.961215 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:00:58.961178 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a3fc99_772f_4628_b9bc_7754ae9e4b1f.slice/crio-dfc13d4a46120396000cbed5ee786071e2ae730e2e4a2184c7e1cfab5cf0afc5 WatchSource:0}: Error finding container dfc13d4a46120396000cbed5ee786071e2ae730e2e4a2184c7e1cfab5cf0afc5: Status 404 returned error can't find the container with id dfc13d4a46120396000cbed5ee786071e2ae730e2e4a2184c7e1cfab5cf0afc5 Nov 03 21:00:59.120107 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:59.120063 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt"] Nov 03 21:00:59.884874 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:59.884842 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg" event={"ID":"74a3fc99-772f-4628-b9bc-7754ae9e4b1f","Type":"ContainerStarted","Data":"a5fd2c8d623e2a31bfb5b13e8bac667dd83d8e9a709f3bc7520209d28c6bf4e7"} Nov 03 21:00:59.885090 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:59.884894 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg" event={"ID":"74a3fc99-772f-4628-b9bc-7754ae9e4b1f","Type":"ContainerStarted","Data":"dfc13d4a46120396000cbed5ee786071e2ae730e2e4a2184c7e1cfab5cf0afc5"} Nov 03 21:00:59.885090 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:59.884934 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg" Nov 03 21:00:59.886645 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:59.886618 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt" event={"ID":"0c60698d-6705-4cfc-8a08-4246536566b5","Type":"ContainerStarted","Data":"e9812bd244502fd1b362287941ea2d4286cf420d65e38d5b75bb217877484f81"} Nov 03 21:00:59.886773 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:59.886658 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt" event={"ID":"0c60698d-6705-4cfc-8a08-4246536566b5","Type":"ContainerStarted","Data":"006cf4fa724ea4eeef1f3df951b93562b182a39cc1719c4641625ca85a7e99d9"} Nov 03 21:00:59.886773 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:59.886678 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt" Nov 03 21:00:59.995948 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:00:59.995897 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg" podStartSLOduration=1.9958830509999999 podStartE2EDuration="1.995883051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:59.994241817 +0000 UTC m=+1931.552232336" watchObservedRunningTime="2026-11-03 21:00:59.995883051 +0000 UTC m=+1931.553873549" Nov 03 21:01:00.103361 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:00.103290 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt" podStartSLOduration=2.103275143 podStartE2EDuration="2.103275143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:00.10070277 +0000 UTC m=+1931.658693270" watchObservedRunningTime="2026-11-03 21:01:00.103275143 +0000 UTC m=+1931.661265643" Nov 03 21:01:00.643940 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:00.643905 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq"] Nov 03 21:01:00.649784 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:00.649721 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq" Nov 03 21:01:00.702406 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:00.702368 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ddbb89-1468-4f64-9250-3a2b32ecdfb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq\" (UID: \"87ddbb89-1468-4f64-9250-3a2b32ecdfb3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq" Nov 03 21:01:00.702595 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:00.702538 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4xx2\" (UniqueName: \"kubernetes.io/projected/87ddbb89-1468-4f64-9250-3a2b32ecdfb3-kube-api-access-s4xx2\") pod \"payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq\" (UID: \"87ddbb89-1468-4f64-9250-3a2b32ecdfb3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq" Nov 03 21:01:00.703391 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:00.703233 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq"] Nov 03 21:01:00.748357 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:00.744428 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-80cbc0-dockercfg-zjpkh\"" Nov 03 21:01:00.748357 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:00.745097 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-80cbc0\"" Nov 03 21:01:00.848531 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:00.841889 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ddbb89-1468-4f64-9250-3a2b32ecdfb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq\" (UID: \"87ddbb89-1468-4f64-9250-3a2b32ecdfb3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq" Nov 03 21:01:00.848531 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:00.842278 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4xx2\" (UniqueName: \"kubernetes.io/projected/87ddbb89-1468-4f64-9250-3a2b32ecdfb3-kube-api-access-s4xx2\") pod \"payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq\" (UID: \"87ddbb89-1468-4f64-9250-3a2b32ecdfb3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq" Nov 03 21:01:00.848531 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:00.843881 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ddbb89-1468-4f64-9250-3a2b32ecdfb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq\" (UID: \"87ddbb89-1468-4f64-9250-3a2b32ecdfb3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq" Nov 03 21:01:00.857666 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:00.857631 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk"] Nov 03 21:01:00.863148 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:00.863125 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk" Nov 03 21:01:00.877101 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:00.877069 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk"] Nov 03 21:01:00.903015 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:00.902934 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4xx2\" (UniqueName: \"kubernetes.io/projected/87ddbb89-1468-4f64-9250-3a2b32ecdfb3-kube-api-access-s4xx2\") pod \"payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq\" (UID: \"87ddbb89-1468-4f64-9250-3a2b32ecdfb3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq" Nov 03 21:01:00.943998 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:00.943954 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b21981-a7ed-4e7e-8ac0-23ac2927547d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk\" (UID: \"40b21981-a7ed-4e7e-8ac0-23ac2927547d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk" Nov 03 21:01:00.944169 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:00.944029 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgppx\" (UniqueName: \"kubernetes.io/projected/40b21981-a7ed-4e7e-8ac0-23ac2927547d-kube-api-access-zgppx\") pod \"payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk\" (UID: \"40b21981-a7ed-4e7e-8ac0-23ac2927547d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk" Nov 03 21:01:00.968036 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:00.968009 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq" Nov 03 21:01:01.046007 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:01.045962 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b21981-a7ed-4e7e-8ac0-23ac2927547d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk\" (UID: \"40b21981-a7ed-4e7e-8ac0-23ac2927547d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk" Nov 03 21:01:01.046154 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:01.046026 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgppx\" (UniqueName: \"kubernetes.io/projected/40b21981-a7ed-4e7e-8ac0-23ac2927547d-kube-api-access-zgppx\") pod \"payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk\" (UID: \"40b21981-a7ed-4e7e-8ac0-23ac2927547d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk" Nov 03 21:01:01.046879 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:01.046833 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b21981-a7ed-4e7e-8ac0-23ac2927547d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk\" (UID: \"40b21981-a7ed-4e7e-8ac0-23ac2927547d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk" Nov 03 21:01:01.100569 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:01.100544 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgppx\" (UniqueName: \"kubernetes.io/projected/40b21981-a7ed-4e7e-8ac0-23ac2927547d-kube-api-access-zgppx\") pod \"payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk\" (UID: \"40b21981-a7ed-4e7e-8ac0-23ac2927547d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk" Nov 03 21:01:01.181459 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:01.181378 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk" Nov 03 21:01:01.346387 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:01.346357 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq"] Nov 03 21:01:01.350765 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:01:01.350734 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ddbb89_1468_4f64_9250_3a2b32ecdfb3.slice/crio-45ea85dd6e5bc95c83e0e7528b8e58b086d281b1121ebedb2139326e262aebce WatchSource:0}: Error finding container 45ea85dd6e5bc95c83e0e7528b8e58b086d281b1121ebedb2139326e262aebce: Status 404 returned error can't find the container with id 45ea85dd6e5bc95c83e0e7528b8e58b086d281b1121ebedb2139326e262aebce Nov 03 21:01:01.559425 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:01.559390 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk"] Nov 03 21:01:01.563608 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:01:01.563582 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b21981_a7ed_4e7e_8ac0_23ac2927547d.slice/crio-d9ff04bf59f408449f86cf3f0cc37c37e4667bb49b692d234b7243edda94a6f2 WatchSource:0}: Error finding container d9ff04bf59f408449f86cf3f0cc37c37e4667bb49b692d234b7243edda94a6f2: Status 404 returned error can't find the container with id d9ff04bf59f408449f86cf3f0cc37c37e4667bb49b692d234b7243edda94a6f2 Nov 03 21:01:01.908455 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:01.908320 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq" event={"ID":"87ddbb89-1468-4f64-9250-3a2b32ecdfb3","Type":"ContainerStarted","Data":"a45fcf186b093448d6430183e0e8e36f6318ea4aa24d3954a1a1c06d96f4add7"} Nov 03 21:01:01.908455 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:01.908436 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq" event={"ID":"87ddbb89-1468-4f64-9250-3a2b32ecdfb3","Type":"ContainerStarted","Data":"45ea85dd6e5bc95c83e0e7528b8e58b086d281b1121ebedb2139326e262aebce"} Nov 03 21:01:01.908971 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:01.908467 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq" Nov 03 21:01:01.910284 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:01.910259 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk" event={"ID":"40b21981-a7ed-4e7e-8ac0-23ac2927547d","Type":"ContainerStarted","Data":"b3c0c4e1ef9411c0c4d9c570c3e42dc3811d1cce8a88ae8a36bb8c43a6966ea1"} Nov 03 21:01:01.910470 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:01.910300 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk" event={"ID":"40b21981-a7ed-4e7e-8ac0-23ac2927547d","Type":"ContainerStarted","Data":"d9ff04bf59f408449f86cf3f0cc37c37e4667bb49b692d234b7243edda94a6f2"} Nov 03 21:01:01.910470 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:01.910321 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk" Nov 03 21:01:02.017583 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:02.017527 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq" podStartSLOduration=2.017512603 podStartE2EDuration="2.017512603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:02.014918156 +0000 UTC m=+1933.572908656" watchObservedRunningTime="2026-11-03 21:01:02.017512603 +0000 UTC m=+1933.575503102" Nov 03 21:01:02.132353 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:02.125782 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk" podStartSLOduration=2.125764772 podStartE2EDuration="2.125764772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:02.120354886 +0000 UTC m=+1933.678345381" watchObservedRunningTime="2026-11-03 21:01:02.125764772 +0000 UTC m=+1933.683755272" Nov 03 21:01:10.034270 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:10.034236 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5"] Nov 03 21:01:10.040107 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:10.040070 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5" Nov 03 21:01:10.091260 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:10.091224 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5"] Nov 03 21:01:10.136359 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:10.135722 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-20d356-dockercfg-cpw55\"" Nov 03 21:01:10.136573 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:10.136380 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-20d356\"" Nov 03 21:01:10.243399 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:10.239312 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2521a657-32ee-4d9b-a1e6-a319fd952387-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5\" (UID: \"2521a657-32ee-4d9b-a1e6-a319fd952387\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5" Nov 03 21:01:10.243399 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:10.239601 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6zb4\" (UniqueName: \"kubernetes.io/projected/2521a657-32ee-4d9b-a1e6-a319fd952387-kube-api-access-r6zb4\") pod \"payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5\" (UID: \"2521a657-32ee-4d9b-a1e6-a319fd952387\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5" Nov 03 21:01:10.340957 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:10.340920 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2521a657-32ee-4d9b-a1e6-a319fd952387-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5\" (UID: \"2521a657-32ee-4d9b-a1e6-a319fd952387\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5" Nov 03 21:01:10.341140 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:10.341071 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6zb4\" (UniqueName: \"kubernetes.io/projected/2521a657-32ee-4d9b-a1e6-a319fd952387-kube-api-access-r6zb4\") pod \"payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5\" (UID: \"2521a657-32ee-4d9b-a1e6-a319fd952387\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5" Nov 03 21:01:10.341772 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:10.341752 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2521a657-32ee-4d9b-a1e6-a319fd952387-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5\" (UID: \"2521a657-32ee-4d9b-a1e6-a319fd952387\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5" Nov 03 21:01:10.395193 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:10.395162 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6zb4\" (UniqueName: \"kubernetes.io/projected/2521a657-32ee-4d9b-a1e6-a319fd952387-kube-api-access-r6zb4\") pod \"payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5\" (UID: \"2521a657-32ee-4d9b-a1e6-a319fd952387\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5" Nov 03 21:01:10.659504 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:10.659427 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5" Nov 03 21:01:10.903872 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:10.903842 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt" Nov 03 21:01:10.904060 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:10.903911 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg" Nov 03 21:01:11.047522 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:01:11.047484 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2521a657_32ee_4d9b_a1e6_a319fd952387.slice/crio-d68cdd3df3838ee423ed247d4a12c8e71f27683fea729d86961cbe1f992599f0 WatchSource:0}: Error finding container d68cdd3df3838ee423ed247d4a12c8e71f27683fea729d86961cbe1f992599f0: Status 404 returned error can't find the container with id d68cdd3df3838ee423ed247d4a12c8e71f27683fea729d86961cbe1f992599f0 Nov 03 21:01:11.053571 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:11.053546 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5"] Nov 03 21:01:11.983173 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:11.983141 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5" event={"ID":"2521a657-32ee-4d9b-a1e6-a319fd952387","Type":"ContainerStarted","Data":"5151d2cacdea429918cd7bee5438c29082acfbca88913662b61ab739e1ee186f"} Nov 03 21:01:11.983375 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:11.983193 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5" event={"ID":"2521a657-32ee-4d9b-a1e6-a319fd952387","Type":"ContainerStarted","Data":"d68cdd3df3838ee423ed247d4a12c8e71f27683fea729d86961cbe1f992599f0"} Nov 03 21:01:11.983375 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:11.983318 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5" Nov 03 21:01:12.091830 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:12.091771 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5" podStartSLOduration=3.091758412 podStartE2EDuration="3.091758412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:12.090304162 +0000 UTC m=+1943.648294680" watchObservedRunningTime="2026-11-03 21:01:12.091758412 +0000 UTC m=+1943.649748911" Nov 03 21:01:12.921545 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:12.921517 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk" Nov 03 21:01:12.921697 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:12.921584 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq" Nov 03 21:01:13.909055 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:13.909023 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq"] Nov 03 21:01:13.909477 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:13.909239 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" podUID="30161e35-b355-4b5f-8746-92522021c10a" containerName="istio-proxy" containerID="cri-o://778efe557a3ce49aa96899027d7cbadb336384edabe7a881d087471f39c2387c" gracePeriod=30 Nov 03 21:01:18.578048 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:18.578009 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:01:18.578545 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:18.578063 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" podUID="30161e35-b355-4b5f-8746-92522021c10a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:01:19.032797 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.032768 4493 generic.go:356] "Generic (PLEG): container finished" podID="30161e35-b355-4b5f-8746-92522021c10a" containerID="778efe557a3ce49aa96899027d7cbadb336384edabe7a881d087471f39c2387c" exitCode=0 Nov 03 21:01:19.032930 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.032846 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" event={"ID":"30161e35-b355-4b5f-8746-92522021c10a","Type":"ContainerDied","Data":"778efe557a3ce49aa96899027d7cbadb336384edabe7a881d087471f39c2387c"} Nov 03 21:01:19.197632 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.197607 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 21:01:19.268456 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.268430 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-workload-certs\") pod \"30161e35-b355-4b5f-8746-92522021c10a\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " Nov 03 21:01:19.268608 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.268543 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-istio-envoy\") pod \"30161e35-b355-4b5f-8746-92522021c10a\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " Nov 03 21:01:19.268608 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.268563 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30161e35-b355-4b5f-8746-92522021c10a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30161e35-b355-4b5f-8746-92522021c10a-istio-podinfo\") pod \"30161e35-b355-4b5f-8746-92522021c10a\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " Nov 03 21:01:19.268688 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.268649 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-istio-data\") pod \"30161e35-b355-4b5f-8746-92522021c10a\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " Nov 03 21:01:19.268688 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.268675 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30161e35-b355-4b5f-8746-92522021c10a-istio-token\" (UniqueName: \"kubernetes.io/projected/30161e35-b355-4b5f-8746-92522021c10a-istio-token\") pod \"30161e35-b355-4b5f-8746-92522021c10a\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " Nov 03 21:01:19.268791 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.268698 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-workload-certs" pod "30161e35-b355-4b5f-8746-92522021c10a" (UID: "30161e35-b355-4b5f-8746-92522021c10a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:01:19.268791 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.268714 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-workload-socket\") pod \"30161e35-b355-4b5f-8746-92522021c10a\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " Nov 03 21:01:19.268895 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.268808 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30161e35-b355-4b5f-8746-92522021c10a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30161e35-b355-4b5f-8746-92522021c10a-istiod-ca-cert\") pod \"30161e35-b355-4b5f-8746-92522021c10a\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " Nov 03 21:01:19.268895 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.268869 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30161e35-b355-4b5f-8746-92522021c10a-kube-api-access-cvn9h\" (UniqueName: \"kubernetes.io/projected/30161e35-b355-4b5f-8746-92522021c10a-kube-api-access-cvn9h\") pod \"30161e35-b355-4b5f-8746-92522021c10a\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " Nov 03 21:01:19.269042 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.268984 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-credential-socket\") pod \"30161e35-b355-4b5f-8746-92522021c10a\" (UID: \"30161e35-b355-4b5f-8746-92522021c10a\") " Nov 03 21:01:19.269390 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.269005 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-istio-data" pod "30161e35-b355-4b5f-8746-92522021c10a" (UID: "30161e35-b355-4b5f-8746-92522021c10a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:01:19.269390 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.269296 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30161e35-b355-4b5f-8746-92522021c10a-istiod-ca-cert" pod "30161e35-b355-4b5f-8746-92522021c10a" (UID: "30161e35-b355-4b5f-8746-92522021c10a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:01:19.269390 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.269012 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-workload-socket" pod "30161e35-b355-4b5f-8746-92522021c10a" (UID: "30161e35-b355-4b5f-8746-92522021c10a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:01:19.269390 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.269212 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-credential-socket" pod "30161e35-b355-4b5f-8746-92522021c10a" (UID: "30161e35-b355-4b5f-8746-92522021c10a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:01:19.270097 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.270071 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:19.270179 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.270102 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:19.270179 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.270118 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30161e35-b355-4b5f-8746-92522021c10a-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:19.270179 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.270132 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:19.270179 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.270146 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:19.271157 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.271130 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30161e35-b355-4b5f-8746-92522021c10a-istio-token" pod "30161e35-b355-4b5f-8746-92522021c10a" (UID: "30161e35-b355-4b5f-8746-92522021c10a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:19.271157 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.271143 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30161e35-b355-4b5f-8746-92522021c10a-istio-podinfo" pod "30161e35-b355-4b5f-8746-92522021c10a" (UID: "30161e35-b355-4b5f-8746-92522021c10a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:01:19.271311 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.271167 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-istio-envoy" pod "30161e35-b355-4b5f-8746-92522021c10a" (UID: "30161e35-b355-4b5f-8746-92522021c10a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:01:19.271418 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.271357 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30161e35-b355-4b5f-8746-92522021c10a-kube-api-access-cvn9h" pod "30161e35-b355-4b5f-8746-92522021c10a" (UID: "30161e35-b355-4b5f-8746-92522021c10a"). InnerVolumeSpecName "kube-api-access-cvn9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:19.372566 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.372529 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30161e35-b355-4b5f-8746-92522021c10a-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:19.372566 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.372569 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30161e35-b355-4b5f-8746-92522021c10a-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:19.372820 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.372585 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30161e35-b355-4b5f-8746-92522021c10a-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:19.372820 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:19.372600 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvn9h\" (UniqueName: \"kubernetes.io/projected/30161e35-b355-4b5f-8746-92522021c10a-kube-api-access-cvn9h\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:20.042416 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:20.042376 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" event={"ID":"30161e35-b355-4b5f-8746-92522021c10a","Type":"ContainerDied","Data":"0c1fe68b8ec87cd3d9c082370df17d902f41fd58c88d897184ea0b0ff979b560"} Nov 03 21:01:20.042933 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:20.042448 4493 scope.go:122] "RemoveContainer" containerID="778efe557a3ce49aa96899027d7cbadb336384edabe7a881d087471f39c2387c" Nov 03 21:01:20.042933 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:20.042395 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq" Nov 03 21:01:20.247390 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:20.247362 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq"] Nov 03 21:01:20.293521 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:20.293446 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-jj9dq"] Nov 03 21:01:20.563983 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:20.563951 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30161e35-b355-4b5f-8746-92522021c10a" path="/var/lib/kubelet/pods/30161e35-b355-4b5f-8746-92522021c10a/volumes" Nov 03 21:01:21.749548 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:01:21.749516 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:01:22.993460 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:22.993429 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5" Nov 03 21:01:23.330882 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:23.330846 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5"] Nov 03 21:01:23.331541 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:23.331527 4493 cpu_manager.go:414] "containerMap: removing container" podUID="30161e35-b355-4b5f-8746-92522021c10a" containerName="istio-proxy" Nov 03 21:01:23.331599 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:23.331542 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30161e35-b355-4b5f-8746-92522021c10a" containerName="istio-proxy" Nov 03 21:01:23.331679 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:23.331668 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="30161e35-b355-4b5f-8746-92522021c10a" containerName="istio-proxy" Nov 03 21:01:23.336741 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:23.336720 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5" Nov 03 21:01:23.385959 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:23.385930 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-88bc35-serving-cert\"" Nov 03 21:01:23.390021 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:23.389998 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5"] Nov 03 21:01:23.429967 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:23.429933 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-openshift-service-ca\") pod \"maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5\" (UID: \"c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5" Nov 03 21:01:23.430152 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:23.430125 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bd5d\" (UniqueName: \"kubernetes.io/projected/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-kube-api-access-2bd5d\") pod \"maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5\" (UID: \"c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5" Nov 03 21:01:23.430217 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:23.430186 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-maas-api-tls\") pod \"maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5\" (UID: \"c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5" Nov 03 21:01:23.430379 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:23.430355 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-metrics-certs\") pod \"maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5\" (UID: \"c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5" Nov 03 21:01:23.531882 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:23.531852 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-openshift-service-ca\") pod \"maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5\" (UID: \"c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5" Nov 03 21:01:23.532033 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:23.531956 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bd5d\" (UniqueName: \"kubernetes.io/projected/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-kube-api-access-2bd5d\") pod \"maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5\" (UID: \"c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5" Nov 03 21:01:23.532033 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:23.531985 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-maas-api-tls\") pod \"maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5\" (UID: \"c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5" Nov 03 21:01:23.532126 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:23.532042 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-metrics-certs\") pod \"maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5\" (UID: \"c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5" Nov 03 21:01:23.532160 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:01:23.532142 4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-serving-cert: secret "maas-api-e2e-api-a-88bc35-serving-cert" not found Nov 03 21:01:23.532233 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:01:23.532222 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-maas-api-tls podName:c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e nodeName:}" failed. No retries permitted until 2026-11-03 21:01:24.032200717 +0000 UTC m=+1955.590191208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-maas-api-tls") pod "maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5" (UID: "c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e") : secret "maas-api-e2e-api-a-88bc35-serving-cert" not found Nov 03 21:01:23.532544 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:23.532524 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-openshift-service-ca\") pod \"maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5\" (UID: \"c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5" Nov 03 21:01:23.534667 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:23.534649 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-metrics-certs\") pod \"maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5\" (UID: \"c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5" Nov 03 21:01:23.589686 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:23.589592 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bd5d\" (UniqueName: \"kubernetes.io/projected/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-kube-api-access-2bd5d\") pod \"maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5\" (UID: \"c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5" Nov 03 21:01:24.044406 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:24.044262 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-maas-api-tls\") pod \"maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5\" (UID: \"c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5" Nov 03 21:01:24.046977 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:24.046955 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-maas-api-tls\") pod \"maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5\" (UID: \"c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5" Nov 03 21:01:24.250790 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:24.250758 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5" Nov 03 21:01:24.627547 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:24.627474 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5"] Nov 03 21:01:24.631302 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:01:24.631275 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc951b9fe_0a9e_443c_8b3c_d2a2df7fb91e.slice/crio-415f5aa59f5d97cc15f2dfd7210457e76503c842fee4dc79767a64dc5972c344 WatchSource:0}: Error finding container 415f5aa59f5d97cc15f2dfd7210457e76503c842fee4dc79767a64dc5972c344: Status 404 returned error can't find the container with id 415f5aa59f5d97cc15f2dfd7210457e76503c842fee4dc79767a64dc5972c344 Nov 03 21:01:25.080436 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:25.080296 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5" event={"ID":"c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e","Type":"ContainerStarted","Data":"415f5aa59f5d97cc15f2dfd7210457e76503c842fee4dc79767a64dc5972c344"} Nov 03 21:01:26.088171 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:26.088134 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5" event={"ID":"c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e","Type":"ContainerStarted","Data":"dd7f4e2fd3c99926656a0b5f55769233008a94b464cb954e61352bd3c7fe6e14"} Nov 03 21:01:26.088627 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:26.088385 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5" Nov 03 21:01:26.198992 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:26.198933 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5" podStartSLOduration=2.2853451160000002 podStartE2EDuration="3.198845575s" totalImagesPullingTime="913.500459ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:26.195727562 +0000 UTC m=+1957.753718079" watchObservedRunningTime="2026-11-03 21:01:26.198845575 +0000 UTC m=+1957.756836073" Nov 03 21:01:32.100625 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:32.100596 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5" Nov 03 21:01:44.218396 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:44.218361 4493 generic.go:356] "Generic (PLEG): container finished" podID="569b73ab-788e-432e-877d-440a81b56965" containerID="703b96b868da99fdea69d97e11417864b67982ec20823ee0f42978b32279c9d0" exitCode=0 Nov 03 21:01:44.218824 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:44.218435 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" event={"ID":"569b73ab-788e-432e-877d-440a81b56965","Type":"ContainerDied","Data":"703b96b868da99fdea69d97e11417864b67982ec20823ee0f42978b32279c9d0"} Nov 03 21:01:44.218898 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:44.218874 4493 scope.go:122] "RemoveContainer" containerID="703b96b868da99fdea69d97e11417864b67982ec20823ee0f42978b32279c9d0" Nov 03 21:01:44.220701 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:44.220673 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:01:44.220779 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:44.220751 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:01:44.220820 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:44.220785 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:01:45.228263 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:45.228232 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" event={"ID":"569b73ab-788e-432e-877d-440a81b56965","Type":"ContainerStarted","Data":"d12c29e6d0c1bdfc3c82aec1e35b4a6f91a66d5d2174cf66311d729aaceace80"} Nov 03 21:01:45.861345 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:45.861310 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:01:45.862301 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:45.862274 4493 patch_prober.go:29] interesting pod/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 21:01:45.862406 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:45.862373 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" podUID="569b73ab-788e-432e-877d-440a81b56965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 21:01:46.861488 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:46.861456 4493 patch_prober.go:29] interesting pod/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 21:01:46.861880 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:46.861506 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" podUID="569b73ab-788e-432e-877d-440a81b56965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 21:01:47.860666 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:47.860628 4493 patch_prober.go:29] interesting pod/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 21:01:47.860849 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:47.860686 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" podUID="569b73ab-788e-432e-877d-440a81b56965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 21:01:48.861078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:48.861042 4493 patch_prober.go:29] interesting pod/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 21:01:48.861510 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:48.861095 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" podUID="569b73ab-788e-432e-877d-440a81b56965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 21:01:49.348914 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:49.348881 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:01:49.861289 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:49.861257 4493 patch_prober.go:29] interesting pod/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 21:01:49.861770 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:49.861308 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" podUID="569b73ab-788e-432e-877d-440a81b56965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 21:01:50.865802 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:50.865772 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:01:50.866878 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:01:50.866853 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:01:51.745930 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:01:51.745894 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:02:01.480650 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:01.480600 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4"] Nov 03 21:02:01.486390 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:01.486369 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4" Nov 03 21:02:01.535581 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:01.535555 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-e6aaa6-serving-cert\"" Nov 03 21:02:01.586384 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:01.586344 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4"] Nov 03 21:02:01.633840 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:01.633808 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655cdf17-2be9-4e5d-bd3a-03b701349675-maas-api-tls\") pod \"maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4\" (UID: \"655cdf17-2be9-4e5d-bd3a-03b701349675\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4" Nov 03 21:02:01.634190 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:01.634164 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/655cdf17-2be9-4e5d-bd3a-03b701349675-openshift-service-ca\") pod \"maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4\" (UID: \"655cdf17-2be9-4e5d-bd3a-03b701349675\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4" Nov 03 21:02:01.634307 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:01.634286 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655cdf17-2be9-4e5d-bd3a-03b701349675-metrics-certs\") pod \"maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4\" (UID: \"655cdf17-2be9-4e5d-bd3a-03b701349675\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4" Nov 03 21:02:01.634439 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:01.634364 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdk6m\" (UniqueName: \"kubernetes.io/projected/655cdf17-2be9-4e5d-bd3a-03b701349675-kube-api-access-wdk6m\") pod \"maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4\" (UID: \"655cdf17-2be9-4e5d-bd3a-03b701349675\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4" Nov 03 21:02:01.738298 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:01.738207 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655cdf17-2be9-4e5d-bd3a-03b701349675-maas-api-tls\") pod \"maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4\" (UID: \"655cdf17-2be9-4e5d-bd3a-03b701349675\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4" Nov 03 21:02:01.738811 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:01.738784 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/655cdf17-2be9-4e5d-bd3a-03b701349675-openshift-service-ca\") pod \"maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4\" (UID: \"655cdf17-2be9-4e5d-bd3a-03b701349675\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4" Nov 03 21:02:01.739995 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:01.739975 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655cdf17-2be9-4e5d-bd3a-03b701349675-metrics-certs\") pod \"maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4\" (UID: \"655cdf17-2be9-4e5d-bd3a-03b701349675\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4" Nov 03 21:02:01.740557 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:01.739922 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/655cdf17-2be9-4e5d-bd3a-03b701349675-openshift-service-ca\") pod \"maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4\" (UID: \"655cdf17-2be9-4e5d-bd3a-03b701349675\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4" Nov 03 21:02:01.740807 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:02:01.738620 4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-serving-cert: secret "maas-api-e2e-api-a-e6aaa6-serving-cert" not found Nov 03 21:02:01.741079 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:02:01.741037 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/655cdf17-2be9-4e5d-bd3a-03b701349675-maas-api-tls podName:655cdf17-2be9-4e5d-bd3a-03b701349675 nodeName:}" failed. No retries permitted until 2026-11-03 21:02:02.240987035 +0000 UTC m=+1993.798977522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/655cdf17-2be9-4e5d-bd3a-03b701349675-maas-api-tls") pod "maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4" (UID: "655cdf17-2be9-4e5d-bd3a-03b701349675") : secret "maas-api-e2e-api-a-e6aaa6-serving-cert" not found Nov 03 21:02:01.742951 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:01.742913 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdk6m\" (UniqueName: \"kubernetes.io/projected/655cdf17-2be9-4e5d-bd3a-03b701349675-kube-api-access-wdk6m\") pod \"maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4\" (UID: \"655cdf17-2be9-4e5d-bd3a-03b701349675\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4" Nov 03 21:02:01.762823 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:01.760748 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655cdf17-2be9-4e5d-bd3a-03b701349675-metrics-certs\") pod \"maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4\" (UID: \"655cdf17-2be9-4e5d-bd3a-03b701349675\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4" Nov 03 21:02:01.799055 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:01.799014 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdk6m\" (UniqueName: \"kubernetes.io/projected/655cdf17-2be9-4e5d-bd3a-03b701349675-kube-api-access-wdk6m\") pod \"maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4\" (UID: \"655cdf17-2be9-4e5d-bd3a-03b701349675\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4" Nov 03 21:02:02.255600 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:02.255566 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655cdf17-2be9-4e5d-bd3a-03b701349675-maas-api-tls\") pod \"maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4\" (UID: \"655cdf17-2be9-4e5d-bd3a-03b701349675\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4" Nov 03 21:02:02.258221 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:02.258192 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655cdf17-2be9-4e5d-bd3a-03b701349675-maas-api-tls\") pod \"maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4\" (UID: \"655cdf17-2be9-4e5d-bd3a-03b701349675\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4" Nov 03 21:02:02.400358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:02.400297 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4" Nov 03 21:02:02.777241 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:02.777205 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4"] Nov 03 21:02:02.780832 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:02:02.780799 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655cdf17_2be9_4e5d_bd3a_03b701349675.slice/crio-ebf1012374541426c385e1094e9cc30e1dc948064bbe7a54848066747b4c9452 WatchSource:0}: Error finding container ebf1012374541426c385e1094e9cc30e1dc948064bbe7a54848066747b4c9452: Status 404 returned error can't find the container with id ebf1012374541426c385e1094e9cc30e1dc948064bbe7a54848066747b4c9452 Nov 03 21:02:03.353476 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:03.353437 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4" event={"ID":"655cdf17-2be9-4e5d-bd3a-03b701349675","Type":"ContainerStarted","Data":"ebf1012374541426c385e1094e9cc30e1dc948064bbe7a54848066747b4c9452"} Nov 03 21:02:04.363302 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:04.363270 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4" event={"ID":"655cdf17-2be9-4e5d-bd3a-03b701349675","Type":"ContainerStarted","Data":"a546c9919bad4fe0ed256ac0106000294e1bba8b213f1d6cac7095dc5ccc3e5c"} Nov 03 21:02:04.363719 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:04.363558 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4" Nov 03 21:02:04.490185 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:04.483717 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4" podStartSLOduration=2.501862769 podStartE2EDuration="3.483691325s" totalImagesPullingTime="981.828556ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:04.475411674 +0000 UTC m=+1996.033402199" watchObservedRunningTime="2026-11-03 21:02:04.483691325 +0000 UTC m=+1996.041681901" Nov 03 21:02:10.376467 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:10.376433 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4" Nov 03 21:02:15.913414 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:15.913365 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5"] Nov 03 21:02:15.918093 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:15.918068 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" Nov 03 21:02:15.956237 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:15.956208 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5"] Nov 03 21:02:15.979510 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:15.979481 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5\" (UID: \"1ad161f0-c9cb-4341-a7a8-55b24e1c1184\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" Nov 03 21:02:15.979669 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:15.979552 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs2fl\" (UniqueName: \"kubernetes.io/projected/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-kube-api-access-gs2fl\") pod \"maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5\" (UID: \"1ad161f0-c9cb-4341-a7a8-55b24e1c1184\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" Nov 03 21:02:15.979743 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:15.979671 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5\" (UID: \"1ad161f0-c9cb-4341-a7a8-55b24e1c1184\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" Nov 03 21:02:15.979843 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:15.979817 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5\" (UID: \"1ad161f0-c9cb-4341-a7a8-55b24e1c1184\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" Nov 03 21:02:16.008618 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:16.008578 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-80cbc0-serving-cert\"" Nov 03 21:02:16.081575 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:16.081547 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs2fl\" (UniqueName: \"kubernetes.io/projected/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-kube-api-access-gs2fl\") pod \"maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5\" (UID: \"1ad161f0-c9cb-4341-a7a8-55b24e1c1184\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" Nov 03 21:02:16.081758 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:16.081589 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5\" (UID: \"1ad161f0-c9cb-4341-a7a8-55b24e1c1184\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" Nov 03 21:02:16.081758 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:16.081701 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5\" (UID: \"1ad161f0-c9cb-4341-a7a8-55b24e1c1184\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" Nov 03 21:02:16.081908 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:16.081837 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5\" (UID: \"1ad161f0-c9cb-4341-a7a8-55b24e1c1184\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" Nov 03 21:02:16.081979 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:02:16.081945 4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-serving-cert: secret "maas-api-e2e-shared-b-w5-80cbc0-serving-cert" not found Nov 03 21:02:16.082035 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:02:16.082018 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-maas-api-tls podName:1ad161f0-c9cb-4341-a7a8-55b24e1c1184 nodeName:}" failed. No retries permitted until 2026-11-03 21:02:16.582002849 +0000 UTC m=+2008.139993326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-maas-api-tls") pod "maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" (UID: "1ad161f0-c9cb-4341-a7a8-55b24e1c1184") : secret "maas-api-e2e-shared-b-w5-80cbc0-serving-cert" not found Nov 03 21:02:16.082386 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:16.082363 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5\" (UID: \"1ad161f0-c9cb-4341-a7a8-55b24e1c1184\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" Nov 03 21:02:16.109384 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:16.104919 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5\" (UID: \"1ad161f0-c9cb-4341-a7a8-55b24e1c1184\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" Nov 03 21:02:16.141125 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:16.138900 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs2fl\" (UniqueName: \"kubernetes.io/projected/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-kube-api-access-gs2fl\") pod \"maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5\" (UID: \"1ad161f0-c9cb-4341-a7a8-55b24e1c1184\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" Nov 03 21:02:16.605360 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:16.602515 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5\" (UID: \"1ad161f0-c9cb-4341-a7a8-55b24e1c1184\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" Nov 03 21:02:16.610356 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:16.608207 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5\" (UID: \"1ad161f0-c9cb-4341-a7a8-55b24e1c1184\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" Nov 03 21:02:16.832640 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:16.832600 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" Nov 03 21:02:17.238451 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:17.238421 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5"] Nov 03 21:02:17.240598 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:02:17.240575 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad161f0_c9cb_4341_a7a8_55b24e1c1184.slice/crio-17aefc2025642dfccc9ce8e7c45a13a2c638594ae39087920585dfc165d60337 WatchSource:0}: Error finding container 17aefc2025642dfccc9ce8e7c45a13a2c638594ae39087920585dfc165d60337: Status 404 returned error can't find the container with id 17aefc2025642dfccc9ce8e7c45a13a2c638594ae39087920585dfc165d60337 Nov 03 21:02:17.454617 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:17.454578 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" event={"ID":"1ad161f0-c9cb-4341-a7a8-55b24e1c1184","Type":"ContainerStarted","Data":"17aefc2025642dfccc9ce8e7c45a13a2c638594ae39087920585dfc165d60337"} Nov 03 21:02:18.467313 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:18.467194 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" event={"ID":"1ad161f0-c9cb-4341-a7a8-55b24e1c1184","Type":"ContainerStarted","Data":"08d5558ead08ca292eeccf374a629f393c6df3458db9f5df9c2ba2b556d3524b"} Nov 03 21:02:18.467313 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:18.467298 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" Nov 03 21:02:18.579129 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:18.579073 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" podStartSLOduration=2.668029131 podStartE2EDuration="3.579052884s" totalImagesPullingTime="911.023753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:18.575208736 +0000 UTC m=+2010.133199267" watchObservedRunningTime="2026-11-03 21:02:18.579052884 +0000 UTC m=+2010.137043383" Nov 03 21:02:21.746039 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:02:21.746004 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:02:23.508088 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:23.508056 4493 generic.go:356] "Generic (PLEG): container finished" podID="8c88cfbc-20ad-47cb-8138-cabbaf57eb1d" containerID="398ad7ddae6b1b89e09731f3562f8648ed7f2a0e9e2abbb4158b4db0369da269" exitCode=0 Nov 03 21:02:23.508559 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:23.508133 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" event={"ID":"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d","Type":"ContainerDied","Data":"398ad7ddae6b1b89e09731f3562f8648ed7f2a0e9e2abbb4158b4db0369da269"} Nov 03 21:02:23.508614 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:23.508600 4493 scope.go:122] "RemoveContainer" containerID="398ad7ddae6b1b89e09731f3562f8648ed7f2a0e9e2abbb4158b4db0369da269" Nov 03 21:02:23.510413 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:23.510387 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:02:23.510476 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:23.510459 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:02:23.510513 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:23.510493 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:02:24.481889 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:24.481864 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" Nov 03 21:02:24.517259 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:24.517227 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" event={"ID":"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d","Type":"ContainerStarted","Data":"0910a8d7214023a76a1f97090bca3f3684262b9f288d63add93c6b5b39f8ed95"} Nov 03 21:02:25.359922 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:25.359890 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 21:02:25.361049 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:25.361018 4493 patch_prober.go:29] interesting pod/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 21:02:25.361187 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:25.361070 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" podUID="8c88cfbc-20ad-47cb-8138-cabbaf57eb1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 21:02:26.360757 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:26.360720 4493 patch_prober.go:29] interesting pod/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 21:02:26.361176 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:26.360773 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" podUID="8c88cfbc-20ad-47cb-8138-cabbaf57eb1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 21:02:27.359931 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:27.359894 4493 patch_prober.go:29] interesting pod/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 21:02:27.360123 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:27.359960 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" podUID="8c88cfbc-20ad-47cb-8138-cabbaf57eb1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 21:02:28.359977 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:28.359943 4493 patch_prober.go:29] interesting pod/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 21:02:28.360413 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:28.359997 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" podUID="8c88cfbc-20ad-47cb-8138-cabbaf57eb1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 21:02:29.204360 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:29.201443 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 21:02:29.360433 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:29.360403 4493 patch_prober.go:29] interesting pod/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 21:02:29.360814 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:29.360453 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" podUID="8c88cfbc-20ad-47cb-8138-cabbaf57eb1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 21:02:30.364575 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:30.364546 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 21:02:30.365669 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:30.365650 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 21:02:50.811726 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:50.811691 4493 generic.go:356] "Generic (PLEG): container finished" podID="46f74fa9-7638-44fa-b303-813d37808176" containerID="389ff94293cd7d3a448630714c20f13a9699dd5cf4ac30bc030c18b5aad79b94" exitCode=0 Nov 03 21:02:50.812216 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:50.811759 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" event={"ID":"46f74fa9-7638-44fa-b303-813d37808176","Type":"ContainerDied","Data":"389ff94293cd7d3a448630714c20f13a9699dd5cf4ac30bc030c18b5aad79b94"} Nov 03 21:02:50.812298 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:50.812278 4493 scope.go:122] "RemoveContainer" containerID="389ff94293cd7d3a448630714c20f13a9699dd5cf4ac30bc030c18b5aad79b94" Nov 03 21:02:50.814076 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:50.814051 4493 generic.go:356] "Generic (PLEG): container finished" podID="2c81f07e-ff42-4534-856a-01f0069a8cb2" containerID="61ea581538c24a83c09a29ac1953b83ee2007762b6e3f0cd1167d00aa0ed7b59" exitCode=0 Nov 03 21:02:50.814217 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:50.814117 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" event={"ID":"2c81f07e-ff42-4534-856a-01f0069a8cb2","Type":"ContainerDied","Data":"61ea581538c24a83c09a29ac1953b83ee2007762b6e3f0cd1167d00aa0ed7b59"} Nov 03 21:02:50.814291 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:50.814202 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:02:50.814291 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:50.814275 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:02:50.814444 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:50.814310 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:02:50.814573 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:50.814554 4493 scope.go:122] "RemoveContainer" containerID="61ea581538c24a83c09a29ac1953b83ee2007762b6e3f0cd1167d00aa0ed7b59" Nov 03 21:02:50.816619 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:50.816586 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:02:50.816725 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:50.816673 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:02:50.816725 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:50.816715 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:02:50.818644 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:50.818626 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:02:51.746413 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:02:51.746378 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:02:51.847518 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:51.847475 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" event={"ID":"46f74fa9-7638-44fa-b303-813d37808176","Type":"ContainerStarted","Data":"86658c8dbf134c8a6d69b430266834e285da6350c2c29582a1cae6a4a6554f10"} Nov 03 21:02:51.853152 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:51.850025 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" event={"ID":"2c81f07e-ff42-4534-856a-01f0069a8cb2","Type":"ContainerStarted","Data":"a366ad942960a4fbe767b18240df00480315a431233f8e84c74ad73f79bd837c"} Nov 03 21:02:52.004418 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:52.000762 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:02:52.004418 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:52.000904 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:02:52.004418 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:52.000952 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" podUID="2c81f07e-ff42-4534-856a-01f0069a8cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 21:02:52.212736 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:52.212698 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:02:52.216796 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:52.216452 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 03 21:02:52.216796 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:52.216515 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" podUID="46f74fa9-7638-44fa-b303-813d37808176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 03 21:02:52.991360 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:52.987671 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:02:52.991360 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:52.987730 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" podUID="2c81f07e-ff42-4534-856a-01f0069a8cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 21:02:53.223364 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:53.213634 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 03 21:02:53.223364 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:53.213695 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" podUID="46f74fa9-7638-44fa-b303-813d37808176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 03 21:02:54.007359 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:54.003747 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:02:54.007359 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:54.003823 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" podUID="2c81f07e-ff42-4534-856a-01f0069a8cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 21:02:54.228766 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:54.228707 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 03 21:02:54.229059 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:54.229031 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" podUID="46f74fa9-7638-44fa-b303-813d37808176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 03 21:02:54.987724 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:54.987686 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:02:54.988021 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:54.987993 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" podUID="2c81f07e-ff42-4534-856a-01f0069a8cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 21:02:55.211084 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:55.211047 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 03 21:02:55.211744 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:55.211710 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" podUID="46f74fa9-7638-44fa-b303-813d37808176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 03 21:02:55.991361 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:55.988308 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:02:55.991361 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:55.988413 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" podUID="2c81f07e-ff42-4534-856a-01f0069a8cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 21:02:56.213363 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:56.212444 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 03 21:02:56.213363 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:56.212506 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" podUID="46f74fa9-7638-44fa-b303-813d37808176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 03 21:02:56.746360 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:56.745527 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:02:56.987728 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:56.987691 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:02:56.987995 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:56.987966 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" podUID="2c81f07e-ff42-4534-856a-01f0069a8cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 21:02:57.217360 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:57.214656 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 03 21:02:57.217360 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:57.214720 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" podUID="46f74fa9-7638-44fa-b303-813d37808176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 03 21:02:57.977810 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:57.977778 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq"] Nov 03 21:02:57.981810 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:57.981791 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:57.989535 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:57.989514 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:02:57.991248 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:57.991224 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:02:58.033978 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.033945 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istiod-ca-cert\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.034099 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.034051 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-podinfo\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.034219 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.034201 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-envoy\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.034470 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.034313 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-data\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.034470 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.034385 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftcg6\" (UniqueName: \"kubernetes.io/projected/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-kube-api-access-ftcg6\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.034744 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.034720 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-credential-socket\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.034837 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.034822 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-workload-certs\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.034900 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.034872 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-token\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.034971 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.034927 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-workload-socket\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.036087 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.036064 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c571ef-openshift-default-dockercfg-pqrkm\"" Nov 03 21:02:58.088451 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.088411 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq"] Nov 03 21:02:58.136568 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.136534 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-envoy\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.136713 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.136614 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-data\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.136713 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.136636 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftcg6\" (UniqueName: \"kubernetes.io/projected/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-kube-api-access-ftcg6\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.136713 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.136690 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-credential-socket\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.136898 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.136734 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-workload-certs\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.136898 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.136762 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-token\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.137018 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.136896 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-workload-socket\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.137018 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.136982 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istiod-ca-cert\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.137136 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.137059 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-data\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.137136 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.137082 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-podinfo\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.137136 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.137108 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-workload-certs\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.137265 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.137182 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-credential-socket\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.137301 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.137259 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-workload-socket\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.137933 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.137903 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istiod-ca-cert\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.139459 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.139413 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-envoy\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.140689 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.140037 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-podinfo\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.191012 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.190963 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-token\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.191204 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.191108 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftcg6\" (UniqueName: \"kubernetes.io/projected/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-kube-api-access-ftcg6\") pod \"e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.215328 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.215298 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:02:58.215680 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.215654 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:02:58.216521 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.216499 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:02:58.298871 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.298833 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:58.749640 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.749563 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq"] Nov 03 21:02:58.756550 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:02:58.754397 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21488ac9_bfba_4bb5_ba76_8c76c7c9fa21.slice/crio-9ff26beb3da47bfc02d0f087732fc3ca21bdba728917acd5df9b1486814e12ca WatchSource:0}: Error finding container 9ff26beb3da47bfc02d0f087732fc3ca21bdba728917acd5df9b1486814e12ca: Status 404 returned error can't find the container with id 9ff26beb3da47bfc02d0f087732fc3ca21bdba728917acd5df9b1486814e12ca Nov 03 21:02:58.759257 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.759218 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:02:58.759388 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.759326 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:02:58.759463 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.759417 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:02:58.928666 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.928622 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" event={"ID":"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21","Type":"ContainerStarted","Data":"2ebfbf22f684f438ec47b8de4b01b53a8c35ff0ebe31c09ab257425d29f92056"} Nov 03 21:02:58.928850 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:58.928690 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" event={"ID":"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21","Type":"ContainerStarted","Data":"9ff26beb3da47bfc02d0f087732fc3ca21bdba728917acd5df9b1486814e12ca"} Nov 03 21:02:59.039377 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:59.039248 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" podStartSLOduration=2.039230342 podStartE2EDuration="2.039230342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:59.037819832 +0000 UTC m=+2050.595810331" watchObservedRunningTime="2026-11-03 21:02:59.039230342 +0000 UTC m=+2050.597220842" Nov 03 21:02:59.299996 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:59.299964 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:59.306484 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:59.306457 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:59.939999 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:59.939968 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:02:59.941216 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:02:59.941189 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:03:21.777366 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:03:21.769305 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:03:39.109507 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.109472 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz"] Nov 03 21:03:39.114729 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.114701 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.157612 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.157576 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-data\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.157612 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.157613 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-token\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.157822 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.157637 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-workload-certs\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.157871 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.157819 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-podinfo\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.157949 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.157936 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-credential-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.158031 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.158008 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istiod-ca-cert\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.158112 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.158099 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-workload-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.158175 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.158161 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-envoy\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.158247 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.158218 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpn9s\" (UniqueName: \"kubernetes.io/projected/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-kube-api-access-qpn9s\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.164491 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.164469 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-20d356-openshift-default-dockercfg-fndvt\"" Nov 03 21:03:39.217865 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.217831 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz"] Nov 03 21:03:39.259631 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.259591 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-envoy\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.259806 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.259666 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpn9s\" (UniqueName: \"kubernetes.io/projected/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-kube-api-access-qpn9s\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.259806 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.259726 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-data\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.259806 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.259760 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-token\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.259806 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.259796 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-workload-certs\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.259952 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.259927 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-podinfo\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.260095 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.260081 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-credential-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.260132 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.260112 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istiod-ca-cert\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.260200 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.260187 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-workload-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.260727 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.260704 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-workload-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.261307 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.261282 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-workload-certs\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.261801 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.261783 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-credential-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.261920 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.261899 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-data\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.262568 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.262542 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istiod-ca-cert\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.263020 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.263001 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-envoy\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.264163 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.264145 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-podinfo\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.316692 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.316621 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-token\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.316874 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.316841 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpn9s\" (UniqueName: \"kubernetes.io/projected/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-kube-api-access-qpn9s\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.430693 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.430615 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:39.859397 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.859363 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz"] Nov 03 21:03:39.863556 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:03:39.863524 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda50088fd_39b0_4118_8f90_c8e6b7fb3dd0.slice/crio-038a5c11aca8b0c352e68c513dfe3ac47cb42c07e8908a2b000f7a238e4caa0b WatchSource:0}: Error finding container 038a5c11aca8b0c352e68c513dfe3ac47cb42c07e8908a2b000f7a238e4caa0b: Status 404 returned error can't find the container with id 038a5c11aca8b0c352e68c513dfe3ac47cb42c07e8908a2b000f7a238e4caa0b Nov 03 21:03:39.866819 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.866786 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:03:39.866904 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.866853 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:03:39.866949 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:39.866902 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:03:40.362296 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:40.362257 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" event={"ID":"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0","Type":"ContainerStarted","Data":"bfad999a5615d928e6071fddaabe2b24a682bc68d423453653c33f8110c60a3e"} Nov 03 21:03:40.362854 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:40.362320 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" event={"ID":"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0","Type":"ContainerStarted","Data":"038a5c11aca8b0c352e68c513dfe3ac47cb42c07e8908a2b000f7a238e4caa0b"} Nov 03 21:03:40.430884 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:40.430853 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:40.432582 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:40.432556 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:03:40.432694 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:40.432607 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" podUID="a50088fd-39b0-4118-8f90-c8e6b7fb3dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 03 21:03:40.477549 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:40.477500 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" podStartSLOduration=1.477482507 podStartE2EDuration="1.477482507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:40.473804673 +0000 UTC m=+2092.031795207" watchObservedRunningTime="2026-11-03 21:03:40.477482507 +0000 UTC m=+2092.035473007" Nov 03 21:03:41.431074 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:41.431037 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:03:41.431518 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:41.431108 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" podUID="a50088fd-39b0-4118-8f90-c8e6b7fb3dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 03 21:03:42.431003 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:42.430969 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:03:42.431166 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:42.431026 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" podUID="a50088fd-39b0-4118-8f90-c8e6b7fb3dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 03 21:03:43.431970 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:43.431936 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:03:43.432392 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:43.432006 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" podUID="a50088fd-39b0-4118-8f90-c8e6b7fb3dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 03 21:03:44.431908 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:44.431875 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:03:44.432070 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:44.431932 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" podUID="a50088fd-39b0-4118-8f90-c8e6b7fb3dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 03 21:03:45.431858 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:45.431824 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:03:45.432115 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:45.431886 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" podUID="a50088fd-39b0-4118-8f90-c8e6b7fb3dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 03 21:03:46.435899 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:46.435870 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:46.436351 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:46.436320 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:46.437091 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:46.437070 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:03:47.235725 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:47.235695 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8"] Nov 03 21:03:47.235917 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:47.235884 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" podUID="a13da4c7-1a94-4992-b4de-b88846e62f21" containerName="istio-proxy" containerID="cri-o://da09b39a2849891ace3dc3df19a1c5cabc72cc3f5f307bf6982cfa097ccd4b02" gracePeriod=30 Nov 03 21:03:47.284279 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:47.284240 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv"] Nov 03 21:03:47.284599 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:47.284566 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" podUID="14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba" containerName="istio-proxy" containerID="cri-o://7c69dfcb738986768ace5780daba2d9bbe8f77170fbff8d768cc7acffbe59263" gracePeriod=30 Nov 03 21:03:48.970788 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:48.970758 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 21:03:48.970788 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:48.970773 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 21:03:49.290870 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:49.290800 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:03:49.291018 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:49.290867 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" podUID="14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:03:51.746714 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:03:51.746678 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:03:52.117081 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.117048 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:03:52.117278 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.117104 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" podUID="a13da4c7-1a94-4992-b4de-b88846e62f21" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:03:52.502521 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.502486 4493 generic.go:356] "Generic (PLEG): container finished" podID="a13da4c7-1a94-4992-b4de-b88846e62f21" containerID="da09b39a2849891ace3dc3df19a1c5cabc72cc3f5f307bf6982cfa097ccd4b02" exitCode=0 Nov 03 21:03:52.502681 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.502557 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" event={"ID":"a13da4c7-1a94-4992-b4de-b88846e62f21","Type":"ContainerDied","Data":"da09b39a2849891ace3dc3df19a1c5cabc72cc3f5f307bf6982cfa097ccd4b02"} Nov 03 21:03:52.504548 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.504522 4493 generic.go:356] "Generic (PLEG): container finished" podID="14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba" containerID="7c69dfcb738986768ace5780daba2d9bbe8f77170fbff8d768cc7acffbe59263" exitCode=0 Nov 03 21:03:52.504667 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.504588 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" event={"ID":"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba","Type":"ContainerDied","Data":"7c69dfcb738986768ace5780daba2d9bbe8f77170fbff8d768cc7acffbe59263"} Nov 03 21:03:52.578442 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.578415 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 21:03:52.631696 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.631666 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 21:03:52.672939 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.672872 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-token\" (UniqueName: \"kubernetes.io/projected/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-token\") pod \"a13da4c7-1a94-4992-b4de-b88846e62f21\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " Nov 03 21:03:52.673074 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.672953 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-data\") pod \"a13da4c7-1a94-4992-b4de-b88846e62f21\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " Nov 03 21:03:52.673074 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.673028 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-envoy\") pod \"a13da4c7-1a94-4992-b4de-b88846e62f21\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " Nov 03 21:03:52.673165 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.673085 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a13da4c7-1a94-4992-b4de-b88846e62f21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a13da4c7-1a94-4992-b4de-b88846e62f21-istiod-ca-cert\") pod \"a13da4c7-1a94-4992-b4de-b88846e62f21\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " Nov 03 21:03:52.673165 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.673129 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-credential-socket\") pod \"a13da4c7-1a94-4992-b4de-b88846e62f21\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " Nov 03 21:03:52.673241 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.673189 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-workload-socket\") pod \"a13da4c7-1a94-4992-b4de-b88846e62f21\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " Nov 03 21:03:52.673241 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.673233 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-podinfo\") pod \"a13da4c7-1a94-4992-b4de-b88846e62f21\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " Nov 03 21:03:52.673318 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.673274 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-data" pod "a13da4c7-1a94-4992-b4de-b88846e62f21" (UID: "a13da4c7-1a94-4992-b4de-b88846e62f21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:52.673379 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.673361 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-workload-certs\") pod \"a13da4c7-1a94-4992-b4de-b88846e62f21\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " Nov 03 21:03:52.673435 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.673383 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-credential-socket" pod "a13da4c7-1a94-4992-b4de-b88846e62f21" (UID: "a13da4c7-1a94-4992-b4de-b88846e62f21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:52.673492 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.673454 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a13da4c7-1a94-4992-b4de-b88846e62f21-kube-api-access-rnv2s\" (UniqueName: \"kubernetes.io/projected/a13da4c7-1a94-4992-b4de-b88846e62f21-kube-api-access-rnv2s\") pod \"a13da4c7-1a94-4992-b4de-b88846e62f21\" (UID: \"a13da4c7-1a94-4992-b4de-b88846e62f21\") " Nov 03 21:03:52.673492 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.673460 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-workload-socket" pod "a13da4c7-1a94-4992-b4de-b88846e62f21" (UID: "a13da4c7-1a94-4992-b4de-b88846e62f21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:52.673597 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.673503 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13da4c7-1a94-4992-b4de-b88846e62f21-istiod-ca-cert" pod "a13da4c7-1a94-4992-b4de-b88846e62f21" (UID: "a13da4c7-1a94-4992-b4de-b88846e62f21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:52.673692 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.673653 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-workload-certs" pod "a13da4c7-1a94-4992-b4de-b88846e62f21" (UID: "a13da4c7-1a94-4992-b4de-b88846e62f21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:52.674212 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.674185 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:52.674375 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.674216 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a13da4c7-1a94-4992-b4de-b88846e62f21-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:52.674375 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.674231 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:52.674375 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.674245 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:52.674375 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.674257 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:52.675505 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.675483 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-envoy" pod "a13da4c7-1a94-4992-b4de-b88846e62f21" (UID: "a13da4c7-1a94-4992-b4de-b88846e62f21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:52.675782 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.675758 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-podinfo" pod "a13da4c7-1a94-4992-b4de-b88846e62f21" (UID: "a13da4c7-1a94-4992-b4de-b88846e62f21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:03:52.676115 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.676098 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13da4c7-1a94-4992-b4de-b88846e62f21-kube-api-access-rnv2s" pod "a13da4c7-1a94-4992-b4de-b88846e62f21" (UID: "a13da4c7-1a94-4992-b4de-b88846e62f21"). InnerVolumeSpecName "kube-api-access-rnv2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:52.676168 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.676123 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-token" pod "a13da4c7-1a94-4992-b4de-b88846e62f21" (UID: "a13da4c7-1a94-4992-b4de-b88846e62f21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:52.775508 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.775474 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-workload-certs\") pod \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " Nov 03 21:03:52.775997 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.775549 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-workload-socket\") pod \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " Nov 03 21:03:52.775997 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.775722 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-envoy\") pod \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " Nov 03 21:03:52.775997 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.775771 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-token\" (UniqueName: \"kubernetes.io/projected/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-token\") pod \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " Nov 03 21:03:52.775997 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.775832 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-workload-certs" pod "14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba" (UID: "14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:52.775997 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.775943 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-workload-socket" pod "14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba" (UID: "14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:52.776278 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.776257 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istiod-ca-cert\") pod \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " Nov 03 21:03:52.776384 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.776369 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-credential-socket\") pod \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " Nov 03 21:03:52.776463 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.776446 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-podinfo\") pod \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " Nov 03 21:03:52.776522 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.776506 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-data\") pod \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " Nov 03 21:03:52.776576 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.776553 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-kube-api-access-h2b5f\" (UniqueName: \"kubernetes.io/projected/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-kube-api-access-h2b5f\") pod \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\" (UID: \"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba\") " Nov 03 21:03:52.777163 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.776773 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-credential-socket" pod "14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba" (UID: "14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:52.777163 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.776796 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istiod-ca-cert" pod "14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba" (UID: "14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:52.777379 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.777358 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:52.777471 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.777386 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:52.777471 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.777402 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:52.777471 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.777416 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:52.777471 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.777430 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:52.777471 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.777444 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnv2s\" (UniqueName: \"kubernetes.io/projected/a13da4c7-1a94-4992-b4de-b88846e62f21-kube-api-access-rnv2s\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:52.777471 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.777458 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:52.777471 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.777471 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a13da4c7-1a94-4992-b4de-b88846e62f21-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:52.778631 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.778600 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-data" pod "14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba" (UID: "14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:52.778631 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.778618 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-envoy" pod "14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba" (UID: "14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:52.779228 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.779186 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-token" pod "14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba" (UID: "14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:52.779376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.779293 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-podinfo" pod "14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba" (UID: "14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:03:52.779701 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.779676 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-kube-api-access-h2b5f" pod "14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba" (UID: "14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba"). InnerVolumeSpecName "kube-api-access-h2b5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:52.878547 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.878514 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:52.878547 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.878541 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:52.878547 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.878551 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:52.878865 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.878560 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:52.878865 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:52.878569 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2b5f\" (UniqueName: \"kubernetes.io/projected/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba-kube-api-access-h2b5f\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:53.512592 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:53.512519 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" Nov 03 21:03:53.512592 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:53.512553 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv" event={"ID":"14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba","Type":"ContainerDied","Data":"0181cd59e438fa0ed681cce5bd061ca664f6fe760a80f40cbe6d9aa735271f26"} Nov 03 21:03:53.512837 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:53.512621 4493 scope.go:122] "RemoveContainer" containerID="7c69dfcb738986768ace5780daba2d9bbe8f77170fbff8d768cc7acffbe59263" Nov 03 21:03:53.514278 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:53.514257 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" Nov 03 21:03:53.514278 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:53.514276 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8" event={"ID":"a13da4c7-1a94-4992-b4de-b88846e62f21","Type":"ContainerDied","Data":"04a7d4910f7703e76f12585f1fb088147c87a8c3a12ac9b48c285e82aad9d399"} Nov 03 21:03:53.525140 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:53.525119 4493 scope.go:122] "RemoveContainer" containerID="da09b39a2849891ace3dc3df19a1c5cabc72cc3f5f307bf6982cfa097ccd4b02" Nov 03 21:03:53.669286 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:53.669254 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv"] Nov 03 21:03:53.716274 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:53.716248 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-ffgpv"] Nov 03 21:03:53.775172 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:53.775076 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8"] Nov 03 21:03:53.778850 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:53.778824 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-987t8"] Nov 03 21:03:54.562847 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:54.562818 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba" path="/var/lib/kubelet/pods/14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba/volumes" Nov 03 21:03:54.563369 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:03:54.563355 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a13da4c7-1a94-4992-b4de-b88846e62f21" path="/var/lib/kubelet/pods/a13da4c7-1a94-4992-b4de-b88846e62f21/volumes" Nov 03 21:04:00.000313 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:00.000224 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs"] Nov 03 21:04:00.000802 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:00.000519 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" podUID="414267d6-2ce4-4d3d-b584-bb64ef5071f6" containerName="istio-proxy" containerID="cri-o://f308f63a3ac62f0b35ba3dfad95b0e683da3e415244c3278c7f3fa859c2430a8" gracePeriod=30 Nov 03 21:04:03.977621 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:03.977589 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:04:03.977991 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:03.977672 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" podUID="414267d6-2ce4-4d3d-b584-bb64ef5071f6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:04:05.276534 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.276476 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 21:04:05.368674 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.368644 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istiod-ca-cert\") pod \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " Nov 03 21:04:05.368868 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.368690 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/414267d6-2ce4-4d3d-b584-bb64ef5071f6-kube-api-access-276mf\" (UniqueName: \"kubernetes.io/projected/414267d6-2ce4-4d3d-b584-bb64ef5071f6-kube-api-access-276mf\") pod \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " Nov 03 21:04:05.368868 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.368728 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-token\" (UniqueName: \"kubernetes.io/projected/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-token\") pod \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " Nov 03 21:04:05.368868 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.368799 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-workload-certs\") pod \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " Nov 03 21:04:05.368868 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.368846 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-credential-socket\") pod \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " Nov 03 21:04:05.369098 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.368893 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-envoy\") pod \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " Nov 03 21:04:05.369098 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.368935 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-data\") pod \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " Nov 03 21:04:05.369098 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.368980 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-podinfo\") pod \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " Nov 03 21:04:05.369098 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.369080 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-workload-socket\") pod \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\" (UID: \"414267d6-2ce4-4d3d-b584-bb64ef5071f6\") " Nov 03 21:04:05.369455 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.369075 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istiod-ca-cert" pod "414267d6-2ce4-4d3d-b584-bb64ef5071f6" (UID: "414267d6-2ce4-4d3d-b584-bb64ef5071f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:05.369455 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.369190 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-credential-socket" pod "414267d6-2ce4-4d3d-b584-bb64ef5071f6" (UID: "414267d6-2ce4-4d3d-b584-bb64ef5071f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:05.369455 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.369228 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-workload-certs" pod "414267d6-2ce4-4d3d-b584-bb64ef5071f6" (UID: "414267d6-2ce4-4d3d-b584-bb64ef5071f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:05.369455 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.369352 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-data" pod "414267d6-2ce4-4d3d-b584-bb64ef5071f6" (UID: "414267d6-2ce4-4d3d-b584-bb64ef5071f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:05.369745 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.369420 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-workload-socket" pod "414267d6-2ce4-4d3d-b584-bb64ef5071f6" (UID: "414267d6-2ce4-4d3d-b584-bb64ef5071f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:05.370039 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.370016 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:05.370146 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.370043 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:05.370146 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.370059 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:05.370146 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.370073 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:05.370146 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.370086 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:05.371409 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.371383 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414267d6-2ce4-4d3d-b584-bb64ef5071f6-kube-api-access-276mf" pod "414267d6-2ce4-4d3d-b584-bb64ef5071f6" (UID: "414267d6-2ce4-4d3d-b584-bb64ef5071f6"). InnerVolumeSpecName "kube-api-access-276mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:05.371585 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.371566 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-envoy" pod "414267d6-2ce4-4d3d-b584-bb64ef5071f6" (UID: "414267d6-2ce4-4d3d-b584-bb64ef5071f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:05.371711 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.371696 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-token" pod "414267d6-2ce4-4d3d-b584-bb64ef5071f6" (UID: "414267d6-2ce4-4d3d-b584-bb64ef5071f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:05.371795 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.371771 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-podinfo" pod "414267d6-2ce4-4d3d-b584-bb64ef5071f6" (UID: "414267d6-2ce4-4d3d-b584-bb64ef5071f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:04:05.472207 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.472171 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:05.472207 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.472203 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:05.472207 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.472213 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-276mf\" (UniqueName: \"kubernetes.io/projected/414267d6-2ce4-4d3d-b584-bb64ef5071f6-kube-api-access-276mf\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:05.472532 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.472223 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/414267d6-2ce4-4d3d-b584-bb64ef5071f6-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:05.605544 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.605511 4493 generic.go:356] "Generic (PLEG): container finished" podID="414267d6-2ce4-4d3d-b584-bb64ef5071f6" containerID="f308f63a3ac62f0b35ba3dfad95b0e683da3e415244c3278c7f3fa859c2430a8" exitCode=0 Nov 03 21:04:05.605745 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.605552 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" Nov 03 21:04:05.605745 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.605592 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" event={"ID":"414267d6-2ce4-4d3d-b584-bb64ef5071f6","Type":"ContainerDied","Data":"f308f63a3ac62f0b35ba3dfad95b0e683da3e415244c3278c7f3fa859c2430a8"} Nov 03 21:04:05.605745 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.605645 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs" event={"ID":"414267d6-2ce4-4d3d-b584-bb64ef5071f6","Type":"ContainerDied","Data":"71079a6f27e023b1faec7e22db954e483bcd9a5d0e1aeec140ea69fe268f34d8"} Nov 03 21:04:05.605745 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.605666 4493 scope.go:122] "RemoveContainer" containerID="f308f63a3ac62f0b35ba3dfad95b0e683da3e415244c3278c7f3fa859c2430a8" Nov 03 21:04:05.617031 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.617015 4493 scope.go:122] "RemoveContainer" containerID="f308f63a3ac62f0b35ba3dfad95b0e683da3e415244c3278c7f3fa859c2430a8" Nov 03 21:04:05.617316 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:04:05.617293 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f308f63a3ac62f0b35ba3dfad95b0e683da3e415244c3278c7f3fa859c2430a8\": container with ID starting with f308f63a3ac62f0b35ba3dfad95b0e683da3e415244c3278c7f3fa859c2430a8 not found: ID does not exist" containerID="f308f63a3ac62f0b35ba3dfad95b0e683da3e415244c3278c7f3fa859c2430a8" Nov 03 21:04:05.617444 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.617324 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f308f63a3ac62f0b35ba3dfad95b0e683da3e415244c3278c7f3fa859c2430a8"} err="failed to get container status \"f308f63a3ac62f0b35ba3dfad95b0e683da3e415244c3278c7f3fa859c2430a8\": rpc error: code = NotFound desc = could not find container \"f308f63a3ac62f0b35ba3dfad95b0e683da3e415244c3278c7f3fa859c2430a8\": container with ID starting with f308f63a3ac62f0b35ba3dfad95b0e683da3e415244c3278c7f3fa859c2430a8 not found: ID does not exist" Nov 03 21:04:05.761637 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.761600 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs"] Nov 03 21:04:05.765282 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:05.765253 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8fncs"] Nov 03 21:04:06.089809 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:06.089778 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j"] Nov 03 21:04:06.201647 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:06.201612 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs"] Nov 03 21:04:06.565705 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:06.565656 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="414267d6-2ce4-4d3d-b584-bb64ef5071f6" path="/var/lib/kubelet/pods/414267d6-2ce4-4d3d-b584-bb64ef5071f6/volumes" Nov 03 21:04:09.340228 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.340190 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw"] Nov 03 21:04:09.340894 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.340875 4493 cpu_manager.go:414] "containerMap: removing container" podUID="a13da4c7-1a94-4992-b4de-b88846e62f21" containerName="istio-proxy" Nov 03 21:04:09.340894 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.340892 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a13da4c7-1a94-4992-b4de-b88846e62f21" containerName="istio-proxy" Nov 03 21:04:09.341062 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.340914 4493 cpu_manager.go:414] "containerMap: removing container" podUID="414267d6-2ce4-4d3d-b584-bb64ef5071f6" containerName="istio-proxy" Nov 03 21:04:09.341062 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.340921 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="414267d6-2ce4-4d3d-b584-bb64ef5071f6" containerName="istio-proxy" Nov 03 21:04:09.341062 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.340938 4493 cpu_manager.go:414] "containerMap: removing container" podUID="14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba" containerName="istio-proxy" Nov 03 21:04:09.341062 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.340946 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba" containerName="istio-proxy" Nov 03 21:04:09.341203 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.341114 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a13da4c7-1a94-4992-b4de-b88846e62f21" containerName="istio-proxy" Nov 03 21:04:09.341203 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.341135 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="414267d6-2ce4-4d3d-b584-bb64ef5071f6" containerName="istio-proxy" Nov 03 21:04:09.341203 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.341147 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="14a14d5e-7956-4ffe-9a82-3ad0de7cb4ba" containerName="istio-proxy" Nov 03 21:04:09.345251 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.345232 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.447998 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.447962 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw"] Nov 03 21:04:09.543526 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.543490 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-token\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.543708 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.543543 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b80b2f-d192-42d5-9657-4d5084cdb967-istiod-ca-cert\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.543793 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.543773 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-workload-certs\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.543834 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.543813 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-podinfo\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.543874 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.543834 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-workload-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.543925 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.543903 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-credential-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.544027 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.544015 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-envoy\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.544083 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.544073 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-data\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.544133 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.544117 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vjh5\" (UniqueName: \"kubernetes.io/projected/08b80b2f-d192-42d5-9657-4d5084cdb967-kube-api-access-7vjh5\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.644988 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.644912 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-envoy\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.644988 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.644979 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-data\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.645207 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.644998 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vjh5\" (UniqueName: \"kubernetes.io/projected/08b80b2f-d192-42d5-9657-4d5084cdb967-kube-api-access-7vjh5\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.645207 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.645026 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-token\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.645207 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.645055 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b80b2f-d192-42d5-9657-4d5084cdb967-istiod-ca-cert\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.645443 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.645234 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-workload-certs\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.645443 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.645275 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-podinfo\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.645443 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.645304 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-workload-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.645443 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.645349 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-credential-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.645596 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.645437 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-data\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.645596 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.645569 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-workload-certs\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.645715 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.645692 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-credential-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.645827 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.645809 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b80b2f-d192-42d5-9657-4d5084cdb967-istiod-ca-cert\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.645866 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.645836 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-workload-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.647557 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.647534 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-envoy\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.648431 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.648410 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-podinfo\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.698314 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.698292 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-token\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.698519 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.698495 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vjh5\" (UniqueName: \"kubernetes.io/projected/08b80b2f-d192-42d5-9657-4d5084cdb967-kube-api-access-7vjh5\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:09.959650 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:09.959490 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:10.417168 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:04:10.417130 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b80b2f_d192_42d5_9657_4d5084cdb967.slice/crio-6bf2cc0a0cc91ed98b42edfcb30dda5a081788847599e8123a8fb6a91675c8eb WatchSource:0}: Error finding container 6bf2cc0a0cc91ed98b42edfcb30dda5a081788847599e8123a8fb6a91675c8eb: Status 404 returned error can't find the container with id 6bf2cc0a0cc91ed98b42edfcb30dda5a081788847599e8123a8fb6a91675c8eb Nov 03 21:04:10.429752 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:10.429711 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:04:10.429990 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:10.429968 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:04:10.430127 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:10.430107 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:04:10.431263 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:10.430840 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw"] Nov 03 21:04:10.648606 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:10.648569 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" event={"ID":"08b80b2f-d192-42d5-9657-4d5084cdb967","Type":"ContainerStarted","Data":"80b183788d53186258439e88bc08ea093f0cd66f76940109eb7fe606692f3e45"} Nov 03 21:04:10.648773 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:10.648636 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" event={"ID":"08b80b2f-d192-42d5-9657-4d5084cdb967","Type":"ContainerStarted","Data":"6bf2cc0a0cc91ed98b42edfcb30dda5a081788847599e8123a8fb6a91675c8eb"} Nov 03 21:04:10.764651 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:10.764521 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" podStartSLOduration=1.7645045910000001 podStartE2EDuration="1.764504591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:10.759459826 +0000 UTC m=+2122.317450338" watchObservedRunningTime="2026-11-03 21:04:10.764504591 +0000 UTC m=+2122.322495089" Nov 03 21:04:10.959795 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:10.959761 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:11.094360 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.092445 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j" podUID="33d6f310-96b9-4029-8f11-179458b5b17c" containerName="payload-processing" containerID="cri-o://09693fe0550ab236a1310a6225b9e0493e01757bb48857dbd81665eddb7efa9d" gracePeriod=25 Nov 03 21:04:11.207363 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.204262 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs" podUID="7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac" containerName="payload-pre-processing" containerID="cri-o://2d1670eafa1fb278031879047fffb9716598bac0a2eeab898efed064fd47b810" gracePeriod=25 Nov 03 21:04:11.431320 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.429830 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j" Nov 03 21:04:11.482201 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.482165 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33d6f310-96b9-4029-8f11-179458b5b17c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33d6f310-96b9-4029-8f11-179458b5b17c-plugins-config-volume\") pod \"33d6f310-96b9-4029-8f11-179458b5b17c\" (UID: \"33d6f310-96b9-4029-8f11-179458b5b17c\") " Nov 03 21:04:11.482403 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.482210 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33d6f310-96b9-4029-8f11-179458b5b17c-kube-api-access-2mjs8\" (UniqueName: \"kubernetes.io/projected/33d6f310-96b9-4029-8f11-179458b5b17c-kube-api-access-2mjs8\") pod \"33d6f310-96b9-4029-8f11-179458b5b17c\" (UID: \"33d6f310-96b9-4029-8f11-179458b5b17c\") " Nov 03 21:04:11.482803 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.482718 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33d6f310-96b9-4029-8f11-179458b5b17c-plugins-config-volume" pod "33d6f310-96b9-4029-8f11-179458b5b17c" (UID: "33d6f310-96b9-4029-8f11-179458b5b17c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:11.482986 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.482935 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33d6f310-96b9-4029-8f11-179458b5b17c-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:11.506358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.485215 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d6f310-96b9-4029-8f11-179458b5b17c-kube-api-access-2mjs8" pod "33d6f310-96b9-4029-8f11-179458b5b17c" (UID: "33d6f310-96b9-4029-8f11-179458b5b17c"). InnerVolumeSpecName "kube-api-access-2mjs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:11.551375 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.549979 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs" Nov 03 21:04:11.608629 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.588040 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac-kube-api-access-djjnd" pod "7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac" (UID: "7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac"). InnerVolumeSpecName "kube-api-access-djjnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:11.612388 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.610444 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac-kube-api-access-djjnd\" (UniqueName: \"kubernetes.io/projected/7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac-kube-api-access-djjnd\") pod \"7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac\" (UID: \"7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac\") " Nov 03 21:04:11.612388 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.610640 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac-plugins-config-volume\") pod \"7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac\" (UID: \"7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac\") " Nov 03 21:04:11.612388 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.611618 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mjs8\" (UniqueName: \"kubernetes.io/projected/33d6f310-96b9-4029-8f11-179458b5b17c-kube-api-access-2mjs8\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:11.612388 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.612377 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac-plugins-config-volume" pod "7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac" (UID: "7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:11.657555 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.657295 4493 generic.go:356] "Generic (PLEG): container finished" podID="7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac" containerID="2d1670eafa1fb278031879047fffb9716598bac0a2eeab898efed064fd47b810" exitCode=0 Nov 03 21:04:11.657555 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.657370 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs" Nov 03 21:04:11.657555 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.657374 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs" event={"ID":"7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac","Type":"ContainerDied","Data":"2d1670eafa1fb278031879047fffb9716598bac0a2eeab898efed064fd47b810"} Nov 03 21:04:11.657555 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.657434 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs" event={"ID":"7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac","Type":"ContainerDied","Data":"04667b727fc2f3373a58e89575a47b6d63d81f76a9b2d7c8c64dee2ab7ad6bd5"} Nov 03 21:04:11.657555 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.657460 4493 scope.go:122] "RemoveContainer" containerID="2d1670eafa1fb278031879047fffb9716598bac0a2eeab898efed064fd47b810" Nov 03 21:04:11.659884 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.659749 4493 generic.go:356] "Generic (PLEG): container finished" podID="33d6f310-96b9-4029-8f11-179458b5b17c" containerID="09693fe0550ab236a1310a6225b9e0493e01757bb48857dbd81665eddb7efa9d" exitCode=0 Nov 03 21:04:11.659884 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.659806 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j" Nov 03 21:04:11.660258 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.659821 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j" event={"ID":"33d6f310-96b9-4029-8f11-179458b5b17c","Type":"ContainerDied","Data":"09693fe0550ab236a1310a6225b9e0493e01757bb48857dbd81665eddb7efa9d"} Nov 03 21:04:11.660926 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.660888 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j" event={"ID":"33d6f310-96b9-4029-8f11-179458b5b17c","Type":"ContainerDied","Data":"29b1b10a422f789f740478fb83ef4d12c9eb06f31f83e1690f9c3aca06ec886a"} Nov 03 21:04:11.674737 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.674723 4493 scope.go:122] "RemoveContainer" containerID="2d1670eafa1fb278031879047fffb9716598bac0a2eeab898efed064fd47b810" Nov 03 21:04:11.675017 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:04:11.674995 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d1670eafa1fb278031879047fffb9716598bac0a2eeab898efed064fd47b810\": container with ID starting with 2d1670eafa1fb278031879047fffb9716598bac0a2eeab898efed064fd47b810 not found: ID does not exist" containerID="2d1670eafa1fb278031879047fffb9716598bac0a2eeab898efed064fd47b810" Nov 03 21:04:11.675069 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.675027 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d1670eafa1fb278031879047fffb9716598bac0a2eeab898efed064fd47b810"} err="failed to get container status \"2d1670eafa1fb278031879047fffb9716598bac0a2eeab898efed064fd47b810\": rpc error: code = NotFound desc = could not find container \"2d1670eafa1fb278031879047fffb9716598bac0a2eeab898efed064fd47b810\": container with ID starting with 2d1670eafa1fb278031879047fffb9716598bac0a2eeab898efed064fd47b810 not found: ID does not exist" Nov 03 21:04:11.675069 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.675043 4493 scope.go:122] "RemoveContainer" containerID="09693fe0550ab236a1310a6225b9e0493e01757bb48857dbd81665eddb7efa9d" Nov 03 21:04:11.716373 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.715579 4493 scope.go:122] "RemoveContainer" containerID="09693fe0550ab236a1310a6225b9e0493e01757bb48857dbd81665eddb7efa9d" Nov 03 21:04:11.716373 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.715864 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djjnd\" (UniqueName: \"kubernetes.io/projected/7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac-kube-api-access-djjnd\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:11.716373 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.715891 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:11.716373 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:04:11.715978 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09693fe0550ab236a1310a6225b9e0493e01757bb48857dbd81665eddb7efa9d\": container with ID starting with 09693fe0550ab236a1310a6225b9e0493e01757bb48857dbd81665eddb7efa9d not found: ID does not exist" containerID="09693fe0550ab236a1310a6225b9e0493e01757bb48857dbd81665eddb7efa9d" Nov 03 21:04:11.716373 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.716009 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09693fe0550ab236a1310a6225b9e0493e01757bb48857dbd81665eddb7efa9d"} err="failed to get container status \"09693fe0550ab236a1310a6225b9e0493e01757bb48857dbd81665eddb7efa9d\": rpc error: code = NotFound desc = could not find container \"09693fe0550ab236a1310a6225b9e0493e01757bb48857dbd81665eddb7efa9d\": container with ID starting with 09693fe0550ab236a1310a6225b9e0493e01757bb48857dbd81665eddb7efa9d not found: ID does not exist" Nov 03 21:04:11.821851 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.821817 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j"] Nov 03 21:04:11.825608 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.825572 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-b6d800-769c87756c-85f7j"] Nov 03 21:04:11.926606 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.926528 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs"] Nov 03 21:04:11.931731 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.931697 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b6d800-697c9fb99f-hnmxs"] Nov 03 21:04:11.961892 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.961857 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 21:04:11.962043 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:11.961916 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" podUID="08b80b2f-d192-42d5-9657-4d5084cdb967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 21:04:12.564673 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:12.564642 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33d6f310-96b9-4029-8f11-179458b5b17c" path="/var/lib/kubelet/pods/33d6f310-96b9-4029-8f11-179458b5b17c/volumes" Nov 03 21:04:12.565126 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:12.565111 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac" path="/var/lib/kubelet/pods/7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac/volumes" Nov 03 21:04:12.960138 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:12.960063 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 21:04:12.960138 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:12.960121 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" podUID="08b80b2f-d192-42d5-9657-4d5084cdb967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 21:04:13.960380 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:13.960317 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 21:04:13.960770 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:13.960396 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" podUID="08b80b2f-d192-42d5-9657-4d5084cdb967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 21:04:14.960419 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:14.960383 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 21:04:14.960889 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:14.960443 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" podUID="08b80b2f-d192-42d5-9657-4d5084cdb967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 21:04:15.924352 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:15.924284 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5"] Nov 03 21:04:15.960378 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:15.960347 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 21:04:15.960549 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:15.960406 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" podUID="08b80b2f-d192-42d5-9657-4d5084cdb967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 21:04:16.965292 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:16.965262 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:16.965839 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:16.965745 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:16.966520 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:16.966494 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:04:20.606029 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:20.605993 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj"] Nov 03 21:04:20.711958 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:20.711526 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt"] Nov 03 21:04:20.925721 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:20.925602 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5" podUID="2521a657-32ee-4d9b-a1e6-a319fd952387" containerName="payload-pre-processing" containerID="cri-o://5151d2cacdea429918cd7bee5438c29082acfbca88913662b61ab739e1ee186f" gracePeriod=25 Nov 03 21:04:21.256188 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:21.256159 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5" Nov 03 21:04:21.362513 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:21.362477 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2521a657-32ee-4d9b-a1e6-a319fd952387-kube-api-access-r6zb4\" (UniqueName: \"kubernetes.io/projected/2521a657-32ee-4d9b-a1e6-a319fd952387-kube-api-access-r6zb4\") pod \"2521a657-32ee-4d9b-a1e6-a319fd952387\" (UID: \"2521a657-32ee-4d9b-a1e6-a319fd952387\") " Nov 03 21:04:21.362716 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:21.362567 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2521a657-32ee-4d9b-a1e6-a319fd952387-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2521a657-32ee-4d9b-a1e6-a319fd952387-plugins-config-volume\") pod \"2521a657-32ee-4d9b-a1e6-a319fd952387\" (UID: \"2521a657-32ee-4d9b-a1e6-a319fd952387\") " Nov 03 21:04:21.363351 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:21.363291 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2521a657-32ee-4d9b-a1e6-a319fd952387-plugins-config-volume" pod "2521a657-32ee-4d9b-a1e6-a319fd952387" (UID: "2521a657-32ee-4d9b-a1e6-a319fd952387"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:21.365421 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:21.365393 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2521a657-32ee-4d9b-a1e6-a319fd952387-kube-api-access-r6zb4" pod "2521a657-32ee-4d9b-a1e6-a319fd952387" (UID: "2521a657-32ee-4d9b-a1e6-a319fd952387"). InnerVolumeSpecName "kube-api-access-r6zb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:21.464673 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:21.464646 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6zb4\" (UniqueName: \"kubernetes.io/projected/2521a657-32ee-4d9b-a1e6-a319fd952387-kube-api-access-r6zb4\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:21.464673 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:21.464673 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2521a657-32ee-4d9b-a1e6-a319fd952387-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:21.748507 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:04:21.748466 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:04:21.774697 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:21.774666 4493 generic.go:356] "Generic (PLEG): container finished" podID="2521a657-32ee-4d9b-a1e6-a319fd952387" containerID="5151d2cacdea429918cd7bee5438c29082acfbca88913662b61ab739e1ee186f" exitCode=0 Nov 03 21:04:21.774871 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:21.774712 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5" Nov 03 21:04:21.774871 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:21.774743 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5" event={"ID":"2521a657-32ee-4d9b-a1e6-a319fd952387","Type":"ContainerDied","Data":"5151d2cacdea429918cd7bee5438c29082acfbca88913662b61ab739e1ee186f"} Nov 03 21:04:21.774871 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:21.774798 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5" event={"ID":"2521a657-32ee-4d9b-a1e6-a319fd952387","Type":"ContainerDied","Data":"d68cdd3df3838ee423ed247d4a12c8e71f27683fea729d86961cbe1f992599f0"} Nov 03 21:04:21.774871 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:21.774831 4493 scope.go:122] "RemoveContainer" containerID="5151d2cacdea429918cd7bee5438c29082acfbca88913662b61ab739e1ee186f" Nov 03 21:04:21.785703 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:21.785685 4493 scope.go:122] "RemoveContainer" containerID="5151d2cacdea429918cd7bee5438c29082acfbca88913662b61ab739e1ee186f" Nov 03 21:04:21.785984 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:04:21.785960 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5151d2cacdea429918cd7bee5438c29082acfbca88913662b61ab739e1ee186f\": container with ID starting with 5151d2cacdea429918cd7bee5438c29082acfbca88913662b61ab739e1ee186f not found: ID does not exist" containerID="5151d2cacdea429918cd7bee5438c29082acfbca88913662b61ab739e1ee186f" Nov 03 21:04:21.786040 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:21.785994 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5151d2cacdea429918cd7bee5438c29082acfbca88913662b61ab739e1ee186f"} err="failed to get container status \"5151d2cacdea429918cd7bee5438c29082acfbca88913662b61ab739e1ee186f\": rpc error: code = NotFound desc = could not find container \"5151d2cacdea429918cd7bee5438c29082acfbca88913662b61ab739e1ee186f\": container with ID starting with 5151d2cacdea429918cd7bee5438c29082acfbca88913662b61ab739e1ee186f not found: ID does not exist" Nov 03 21:04:21.933212 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:21.933176 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5"] Nov 03 21:04:21.936770 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:21.936743 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-20d356-67d6675775-sxrs5"] Nov 03 21:04:22.563314 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:22.563276 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2521a657-32ee-4d9b-a1e6-a319fd952387" path="/var/lib/kubelet/pods/2521a657-32ee-4d9b-a1e6-a319fd952387/volumes" Nov 03 21:04:25.606575 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:25.606512 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj" podUID="f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898" containerName="payload-processing" containerID="cri-o://7b2dd6ab4adba0820274b3cd2a930aa6f3e496d48573172d5c93b2736bec85dd" gracePeriod=25 Nov 03 21:04:25.712201 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:25.712129 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt" podUID="d12d909a-e17b-470b-9ba6-164d1dbffba1" containerName="payload-pre-processing" containerID="cri-o://d6e853a62dde0a69ab54e59c5ee0201026b92be49d7db3e85bf172577e797729" gracePeriod=25 Nov 03 21:04:25.820097 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:25.820040 4493 generic.go:356] "Generic (PLEG): container finished" podID="f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898" containerID="7b2dd6ab4adba0820274b3cd2a930aa6f3e496d48573172d5c93b2736bec85dd" exitCode=0 Nov 03 21:04:25.820254 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:25.820168 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj" event={"ID":"f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898","Type":"ContainerDied","Data":"7b2dd6ab4adba0820274b3cd2a930aa6f3e496d48573172d5c93b2736bec85dd"} Nov 03 21:04:25.887538 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:25.887503 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj" Nov 03 21:04:26.009998 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:26.009971 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt" Nov 03 21:04:26.041129 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:26.041101 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898-kube-api-access-rqrw2\" (UniqueName: \"kubernetes.io/projected/f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898-kube-api-access-rqrw2\") pod \"f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898\" (UID: \"f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898\") " Nov 03 21:04:26.041312 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:26.041165 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898-plugins-config-volume\") pod \"f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898\" (UID: \"f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898\") " Nov 03 21:04:26.041620 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:26.041591 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898-plugins-config-volume" pod "f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898" (UID: "f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:26.041941 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:26.041920 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:26.043536 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:26.043516 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898-kube-api-access-rqrw2" pod "f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898" (UID: "f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898"). InnerVolumeSpecName "kube-api-access-rqrw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:26.142876 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:26.142793 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d12d909a-e17b-470b-9ba6-164d1dbffba1-kube-api-access-7krj5\" (UniqueName: \"kubernetes.io/projected/d12d909a-e17b-470b-9ba6-164d1dbffba1-kube-api-access-7krj5\") pod \"d12d909a-e17b-470b-9ba6-164d1dbffba1\" (UID: \"d12d909a-e17b-470b-9ba6-164d1dbffba1\") " Nov 03 21:04:26.143054 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:26.142948 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d12d909a-e17b-470b-9ba6-164d1dbffba1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d12d909a-e17b-470b-9ba6-164d1dbffba1-plugins-config-volume\") pod \"d12d909a-e17b-470b-9ba6-164d1dbffba1\" (UID: \"d12d909a-e17b-470b-9ba6-164d1dbffba1\") " Nov 03 21:04:26.143426 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:26.143396 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12d909a-e17b-470b-9ba6-164d1dbffba1-plugins-config-volume" pod "d12d909a-e17b-470b-9ba6-164d1dbffba1" (UID: "d12d909a-e17b-470b-9ba6-164d1dbffba1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:26.143766 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:26.143738 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqrw2\" (UniqueName: \"kubernetes.io/projected/f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898-kube-api-access-rqrw2\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:26.143887 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:26.143770 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d12d909a-e17b-470b-9ba6-164d1dbffba1-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:26.145665 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:26.145636 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12d909a-e17b-470b-9ba6-164d1dbffba1-kube-api-access-7krj5" pod "d12d909a-e17b-470b-9ba6-164d1dbffba1" (UID: "d12d909a-e17b-470b-9ba6-164d1dbffba1"). InnerVolumeSpecName "kube-api-access-7krj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:26.244824 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:26.244792 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7krj5\" (UniqueName: \"kubernetes.io/projected/d12d909a-e17b-470b-9ba6-164d1dbffba1-kube-api-access-7krj5\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:26.828688 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:26.828594 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj" Nov 03 21:04:26.828688 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:26.828622 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj" event={"ID":"f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898","Type":"ContainerDied","Data":"ab764eb0257655056f357543cfa231c810fa8505447b6fcb823ae353c61b12f7"} Nov 03 21:04:26.829201 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:26.828698 4493 scope.go:122] "RemoveContainer" containerID="7b2dd6ab4adba0820274b3cd2a930aa6f3e496d48573172d5c93b2736bec85dd" Nov 03 21:04:26.830097 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:26.830061 4493 generic.go:356] "Generic (PLEG): container finished" podID="d12d909a-e17b-470b-9ba6-164d1dbffba1" containerID="d6e853a62dde0a69ab54e59c5ee0201026b92be49d7db3e85bf172577e797729" exitCode=0 Nov 03 21:04:26.830191 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:26.830099 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt" event={"ID":"d12d909a-e17b-470b-9ba6-164d1dbffba1","Type":"ContainerDied","Data":"d6e853a62dde0a69ab54e59c5ee0201026b92be49d7db3e85bf172577e797729"} Nov 03 21:04:26.830191 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:26.830164 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt" event={"ID":"d12d909a-e17b-470b-9ba6-164d1dbffba1","Type":"ContainerDied","Data":"8118fa4a1ef21223bc9e89f2a08f45393fc6675806d9168bea4916215690aac3"} Nov 03 21:04:26.830191 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:26.830115 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt" Nov 03 21:04:26.840222 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:26.840202 4493 scope.go:122] "RemoveContainer" containerID="d6e853a62dde0a69ab54e59c5ee0201026b92be49d7db3e85bf172577e797729" Nov 03 21:04:26.850073 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:26.850058 4493 scope.go:122] "RemoveContainer" containerID="d6e853a62dde0a69ab54e59c5ee0201026b92be49d7db3e85bf172577e797729" Nov 03 21:04:26.850386 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:04:26.850364 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6e853a62dde0a69ab54e59c5ee0201026b92be49d7db3e85bf172577e797729\": container with ID starting with d6e853a62dde0a69ab54e59c5ee0201026b92be49d7db3e85bf172577e797729 not found: ID does not exist" containerID="d6e853a62dde0a69ab54e59c5ee0201026b92be49d7db3e85bf172577e797729" Nov 03 21:04:26.850474 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:26.850396 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6e853a62dde0a69ab54e59c5ee0201026b92be49d7db3e85bf172577e797729"} err="failed to get container status \"d6e853a62dde0a69ab54e59c5ee0201026b92be49d7db3e85bf172577e797729\": rpc error: code = NotFound desc = could not find container \"d6e853a62dde0a69ab54e59c5ee0201026b92be49d7db3e85bf172577e797729\": container with ID starting with d6e853a62dde0a69ab54e59c5ee0201026b92be49d7db3e85bf172577e797729 not found: ID does not exist" Nov 03 21:04:26.984847 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:26.984813 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt"] Nov 03 21:04:27.031543 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:27.031513 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-1ec1b6-85cf96f6d7-mw9vt"] Nov 03 21:04:27.092718 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:27.092635 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj"] Nov 03 21:04:27.096109 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:27.096082 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-1ec1b6-5f9bfdbdc7-gcwjj"] Nov 03 21:04:28.563344 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:28.563296 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d12d909a-e17b-470b-9ba6-164d1dbffba1" path="/var/lib/kubelet/pods/d12d909a-e17b-470b-9ba6-164d1dbffba1/volumes" Nov 03 21:04:28.563858 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:04:28.563841 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898" path="/var/lib/kubelet/pods/f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898/volumes" Nov 03 21:04:51.745876 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:04:51.745835 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:05:06.413390 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:06.413356 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5"] Nov 03 21:05:06.413930 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:06.413624 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5" podUID="6a0808b2-7798-4f60-9435-063b72bbdb21" containerName="maas-api" containerID="cri-o://46052c58d0c3ed596a21a51897546db2ecf0735a968a75cc99b8d7d9dad390de" gracePeriod=30 Nov 03 21:05:06.589119 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:06.589083 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk"] Nov 03 21:05:06.690610 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:06.690522 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5" Nov 03 21:05:06.694595 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:06.694569 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq"] Nov 03 21:05:06.748754 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:06.748727 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a0808b2-7798-4f60-9435-063b72bbdb21-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a0808b2-7798-4f60-9435-063b72bbdb21-openshift-service-ca\") pod \"6a0808b2-7798-4f60-9435-063b72bbdb21\" (UID: \"6a0808b2-7798-4f60-9435-063b72bbdb21\") " Nov 03 21:05:06.748896 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:06.748784 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a0808b2-7798-4f60-9435-063b72bbdb21-kube-api-access-rzgmk\" (UniqueName: \"kubernetes.io/projected/6a0808b2-7798-4f60-9435-063b72bbdb21-kube-api-access-rzgmk\") pod \"6a0808b2-7798-4f60-9435-063b72bbdb21\" (UID: \"6a0808b2-7798-4f60-9435-063b72bbdb21\") " Nov 03 21:05:06.748896 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:06.748847 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a0808b2-7798-4f60-9435-063b72bbdb21-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a0808b2-7798-4f60-9435-063b72bbdb21-maas-api-tls\") pod \"6a0808b2-7798-4f60-9435-063b72bbdb21\" (UID: \"6a0808b2-7798-4f60-9435-063b72bbdb21\") " Nov 03 21:05:06.748896 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:06.748866 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a0808b2-7798-4f60-9435-063b72bbdb21-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a0808b2-7798-4f60-9435-063b72bbdb21-metrics-certs\") pod \"6a0808b2-7798-4f60-9435-063b72bbdb21\" (UID: \"6a0808b2-7798-4f60-9435-063b72bbdb21\") " Nov 03 21:05:06.749079 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:06.749057 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a0808b2-7798-4f60-9435-063b72bbdb21-openshift-service-ca" pod "6a0808b2-7798-4f60-9435-063b72bbdb21" (UID: "6a0808b2-7798-4f60-9435-063b72bbdb21"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:06.749434 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:06.749411 4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a0808b2-7798-4f60-9435-063b72bbdb21-openshift-service-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:06.751153 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:06.751131 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0808b2-7798-4f60-9435-063b72bbdb21-maas-api-tls" pod "6a0808b2-7798-4f60-9435-063b72bbdb21" (UID: "6a0808b2-7798-4f60-9435-063b72bbdb21"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:05:06.751258 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:06.751205 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0808b2-7798-4f60-9435-063b72bbdb21-kube-api-access-rzgmk" pod "6a0808b2-7798-4f60-9435-063b72bbdb21" (UID: "6a0808b2-7798-4f60-9435-063b72bbdb21"). InnerVolumeSpecName "kube-api-access-rzgmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:06.751258 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:06.751220 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0808b2-7798-4f60-9435-063b72bbdb21-metrics-certs" pod "6a0808b2-7798-4f60-9435-063b72bbdb21" (UID: "6a0808b2-7798-4f60-9435-063b72bbdb21"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:05:06.850704 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:06.850674 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a0808b2-7798-4f60-9435-063b72bbdb21-maas-api-tls\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:06.850704 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:06.850702 4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a0808b2-7798-4f60-9435-063b72bbdb21-metrics-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:06.850889 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:06.850711 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzgmk\" (UniqueName: \"kubernetes.io/projected/6a0808b2-7798-4f60-9435-063b72bbdb21-kube-api-access-rzgmk\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:07.150778 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:07.150744 4493 generic.go:356] "Generic (PLEG): container finished" podID="6a0808b2-7798-4f60-9435-063b72bbdb21" containerID="46052c58d0c3ed596a21a51897546db2ecf0735a968a75cc99b8d7d9dad390de" exitCode=0 Nov 03 21:05:07.150990 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:07.150785 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5" event={"ID":"6a0808b2-7798-4f60-9435-063b72bbdb21","Type":"ContainerDied","Data":"46052c58d0c3ed596a21a51897546db2ecf0735a968a75cc99b8d7d9dad390de"} Nov 03 21:05:07.150990 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:07.150804 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5" Nov 03 21:05:07.150990 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:07.150833 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5" event={"ID":"6a0808b2-7798-4f60-9435-063b72bbdb21","Type":"ContainerDied","Data":"d656819b3d12ec61c757a0300fed5527a29363433d718728766a745f58459813"} Nov 03 21:05:07.150990 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:07.150856 4493 scope.go:122] "RemoveContainer" containerID="46052c58d0c3ed596a21a51897546db2ecf0735a968a75cc99b8d7d9dad390de" Nov 03 21:05:07.162647 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:07.162630 4493 scope.go:122] "RemoveContainer" containerID="46052c58d0c3ed596a21a51897546db2ecf0735a968a75cc99b8d7d9dad390de" Nov 03 21:05:07.162953 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:05:07.162934 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46052c58d0c3ed596a21a51897546db2ecf0735a968a75cc99b8d7d9dad390de\": container with ID starting with 46052c58d0c3ed596a21a51897546db2ecf0735a968a75cc99b8d7d9dad390de not found: ID does not exist" containerID="46052c58d0c3ed596a21a51897546db2ecf0735a968a75cc99b8d7d9dad390de" Nov 03 21:05:07.163002 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:07.162962 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46052c58d0c3ed596a21a51897546db2ecf0735a968a75cc99b8d7d9dad390de"} err="failed to get container status \"46052c58d0c3ed596a21a51897546db2ecf0735a968a75cc99b8d7d9dad390de\": rpc error: code = NotFound desc = could not find container \"46052c58d0c3ed596a21a51897546db2ecf0735a968a75cc99b8d7d9dad390de\": container with ID starting with 46052c58d0c3ed596a21a51897546db2ecf0735a968a75cc99b8d7d9dad390de not found: ID does not exist" Nov 03 21:05:07.307327 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:07.307267 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5"] Nov 03 21:05:07.311278 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:07.311253 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1ec1b6-c96c5c5d8-kp9z5"] Nov 03 21:05:08.565728 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:08.565696 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a0808b2-7798-4f60-9435-063b72bbdb21" path="/var/lib/kubelet/pods/6a0808b2-7798-4f60-9435-063b72bbdb21/volumes" Nov 03 21:05:11.589994 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:11.589950 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk" podUID="40b21981-a7ed-4e7e-8ac0-23ac2927547d" containerName="payload-processing" containerID="cri-o://b3c0c4e1ef9411c0c4d9c570c3e42dc3811d1cce8a88ae8a36bb8c43a6966ea1" gracePeriod=25 Nov 03 21:05:11.695424 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:11.695320 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq" podUID="87ddbb89-1468-4f64-9250-3a2b32ecdfb3" containerName="payload-pre-processing" containerID="cri-o://a45fcf186b093448d6430183e0e8e36f6318ea4aa24d3954a1a1c06d96f4add7" gracePeriod=25 Nov 03 21:05:11.885945 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:11.885914 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk" Nov 03 21:05:11.940516 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:11.940484 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40b21981-a7ed-4e7e-8ac0-23ac2927547d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b21981-a7ed-4e7e-8ac0-23ac2927547d-plugins-config-volume\") pod \"40b21981-a7ed-4e7e-8ac0-23ac2927547d\" (UID: \"40b21981-a7ed-4e7e-8ac0-23ac2927547d\") " Nov 03 21:05:11.940714 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:11.940540 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b21981-a7ed-4e7e-8ac0-23ac2927547d-kube-api-access-zgppx\" (UniqueName: \"kubernetes.io/projected/40b21981-a7ed-4e7e-8ac0-23ac2927547d-kube-api-access-zgppx\") pod \"40b21981-a7ed-4e7e-8ac0-23ac2927547d\" (UID: \"40b21981-a7ed-4e7e-8ac0-23ac2927547d\") " Nov 03 21:05:11.940923 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:11.940896 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b21981-a7ed-4e7e-8ac0-23ac2927547d-plugins-config-volume" pod "40b21981-a7ed-4e7e-8ac0-23ac2927547d" (UID: "40b21981-a7ed-4e7e-8ac0-23ac2927547d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:11.941356 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:11.941310 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b21981-a7ed-4e7e-8ac0-23ac2927547d-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:11.943063 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:11.943030 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b21981-a7ed-4e7e-8ac0-23ac2927547d-kube-api-access-zgppx" pod "40b21981-a7ed-4e7e-8ac0-23ac2927547d" (UID: "40b21981-a7ed-4e7e-8ac0-23ac2927547d"). InnerVolumeSpecName "kube-api-access-zgppx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:11.963527 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:11.963506 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq" Nov 03 21:05:12.041919 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.041886 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87ddbb89-1468-4f64-9250-3a2b32ecdfb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ddbb89-1468-4f64-9250-3a2b32ecdfb3-plugins-config-volume\") pod \"87ddbb89-1468-4f64-9250-3a2b32ecdfb3\" (UID: \"87ddbb89-1468-4f64-9250-3a2b32ecdfb3\") " Nov 03 21:05:12.042100 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.041943 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87ddbb89-1468-4f64-9250-3a2b32ecdfb3-kube-api-access-s4xx2\" (UniqueName: \"kubernetes.io/projected/87ddbb89-1468-4f64-9250-3a2b32ecdfb3-kube-api-access-s4xx2\") pod \"87ddbb89-1468-4f64-9250-3a2b32ecdfb3\" (UID: \"87ddbb89-1468-4f64-9250-3a2b32ecdfb3\") " Nov 03 21:05:12.042270 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.042245 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ddbb89-1468-4f64-9250-3a2b32ecdfb3-plugins-config-volume" pod "87ddbb89-1468-4f64-9250-3a2b32ecdfb3" (UID: "87ddbb89-1468-4f64-9250-3a2b32ecdfb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:12.042697 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.042615 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ddbb89-1468-4f64-9250-3a2b32ecdfb3-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:12.042697 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.042640 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgppx\" (UniqueName: \"kubernetes.io/projected/40b21981-a7ed-4e7e-8ac0-23ac2927547d-kube-api-access-zgppx\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:12.044283 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.044259 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ddbb89-1468-4f64-9250-3a2b32ecdfb3-kube-api-access-s4xx2" pod "87ddbb89-1468-4f64-9250-3a2b32ecdfb3" (UID: "87ddbb89-1468-4f64-9250-3a2b32ecdfb3"). InnerVolumeSpecName "kube-api-access-s4xx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:12.143739 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.143660 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4xx2\" (UniqueName: \"kubernetes.io/projected/87ddbb89-1468-4f64-9250-3a2b32ecdfb3-kube-api-access-s4xx2\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:12.218432 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.218399 4493 generic.go:356] "Generic (PLEG): container finished" podID="40b21981-a7ed-4e7e-8ac0-23ac2927547d" containerID="b3c0c4e1ef9411c0c4d9c570c3e42dc3811d1cce8a88ae8a36bb8c43a6966ea1" exitCode=0 Nov 03 21:05:12.218595 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.218444 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk" Nov 03 21:05:12.218595 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.218443 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk" event={"ID":"40b21981-a7ed-4e7e-8ac0-23ac2927547d","Type":"ContainerDied","Data":"b3c0c4e1ef9411c0c4d9c570c3e42dc3811d1cce8a88ae8a36bb8c43a6966ea1"} Nov 03 21:05:12.218595 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.218571 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk" event={"ID":"40b21981-a7ed-4e7e-8ac0-23ac2927547d","Type":"ContainerDied","Data":"d9ff04bf59f408449f86cf3f0cc37c37e4667bb49b692d234b7243edda94a6f2"} Nov 03 21:05:12.218725 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.218601 4493 scope.go:122] "RemoveContainer" containerID="b3c0c4e1ef9411c0c4d9c570c3e42dc3811d1cce8a88ae8a36bb8c43a6966ea1" Nov 03 21:05:12.220058 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.220033 4493 generic.go:356] "Generic (PLEG): container finished" podID="87ddbb89-1468-4f64-9250-3a2b32ecdfb3" containerID="a45fcf186b093448d6430183e0e8e36f6318ea4aa24d3954a1a1c06d96f4add7" exitCode=0 Nov 03 21:05:12.220184 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.220091 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq" Nov 03 21:05:12.220184 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.220100 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq" event={"ID":"87ddbb89-1468-4f64-9250-3a2b32ecdfb3","Type":"ContainerDied","Data":"a45fcf186b093448d6430183e0e8e36f6318ea4aa24d3954a1a1c06d96f4add7"} Nov 03 21:05:12.220184 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.220155 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq" event={"ID":"87ddbb89-1468-4f64-9250-3a2b32ecdfb3","Type":"ContainerDied","Data":"45ea85dd6e5bc95c83e0e7528b8e58b086d281b1121ebedb2139326e262aebce"} Nov 03 21:05:12.232876 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.232765 4493 scope.go:122] "RemoveContainer" containerID="b3c0c4e1ef9411c0c4d9c570c3e42dc3811d1cce8a88ae8a36bb8c43a6966ea1" Nov 03 21:05:12.233085 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:05:12.233067 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3c0c4e1ef9411c0c4d9c570c3e42dc3811d1cce8a88ae8a36bb8c43a6966ea1\": container with ID starting with b3c0c4e1ef9411c0c4d9c570c3e42dc3811d1cce8a88ae8a36bb8c43a6966ea1 not found: ID does not exist" containerID="b3c0c4e1ef9411c0c4d9c570c3e42dc3811d1cce8a88ae8a36bb8c43a6966ea1" Nov 03 21:05:12.233164 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.233096 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c0c4e1ef9411c0c4d9c570c3e42dc3811d1cce8a88ae8a36bb8c43a6966ea1"} err="failed to get container status \"b3c0c4e1ef9411c0c4d9c570c3e42dc3811d1cce8a88ae8a36bb8c43a6966ea1\": rpc error: code = NotFound desc = could not find container \"b3c0c4e1ef9411c0c4d9c570c3e42dc3811d1cce8a88ae8a36bb8c43a6966ea1\": container with ID starting with b3c0c4e1ef9411c0c4d9c570c3e42dc3811d1cce8a88ae8a36bb8c43a6966ea1 not found: ID does not exist" Nov 03 21:05:12.233164 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.233120 4493 scope.go:122] "RemoveContainer" containerID="a45fcf186b093448d6430183e0e8e36f6318ea4aa24d3954a1a1c06d96f4add7" Nov 03 21:05:12.242818 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.242801 4493 scope.go:122] "RemoveContainer" containerID="a45fcf186b093448d6430183e0e8e36f6318ea4aa24d3954a1a1c06d96f4add7" Nov 03 21:05:12.243080 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:05:12.243062 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a45fcf186b093448d6430183e0e8e36f6318ea4aa24d3954a1a1c06d96f4add7\": container with ID starting with a45fcf186b093448d6430183e0e8e36f6318ea4aa24d3954a1a1c06d96f4add7 not found: ID does not exist" containerID="a45fcf186b093448d6430183e0e8e36f6318ea4aa24d3954a1a1c06d96f4add7" Nov 03 21:05:12.243128 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.243088 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a45fcf186b093448d6430183e0e8e36f6318ea4aa24d3954a1a1c06d96f4add7"} err="failed to get container status \"a45fcf186b093448d6430183e0e8e36f6318ea4aa24d3954a1a1c06d96f4add7\": rpc error: code = NotFound desc = could not find container \"a45fcf186b093448d6430183e0e8e36f6318ea4aa24d3954a1a1c06d96f4add7\": container with ID starting with a45fcf186b093448d6430183e0e8e36f6318ea4aa24d3954a1a1c06d96f4add7 not found: ID does not exist" Nov 03 21:05:12.376396 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.376366 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk"] Nov 03 21:05:12.379912 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.379884 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-80cbc0-5cd6b65d6f-289wk"] Nov 03 21:05:12.479051 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.478976 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq"] Nov 03 21:05:12.483217 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.483189 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-80cbc0-76dd9bd7ff-5gvmq"] Nov 03 21:05:12.563203 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.563169 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40b21981-a7ed-4e7e-8ac0-23ac2927547d" path="/var/lib/kubelet/pods/40b21981-a7ed-4e7e-8ac0-23ac2927547d/volumes" Nov 03 21:05:12.563667 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.563652 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87ddbb89-1468-4f64-9250-3a2b32ecdfb3" path="/var/lib/kubelet/pods/87ddbb89-1468-4f64-9250-3a2b32ecdfb3/volumes" Nov 03 21:05:12.804597 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.804564 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4"] Nov 03 21:05:12.804959 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:12.804758 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4" podUID="655cdf17-2be9-4e5d-bd3a-03b701349675" containerName="maas-api" containerID="cri-o://a546c9919bad4fe0ed256ac0106000294e1bba8b213f1d6cac7095dc5ccc3e5c" gracePeriod=30 Nov 03 21:05:13.122806 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:13.122774 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4" Nov 03 21:05:13.238452 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:13.238413 4493 generic.go:356] "Generic (PLEG): container finished" podID="655cdf17-2be9-4e5d-bd3a-03b701349675" containerID="a546c9919bad4fe0ed256ac0106000294e1bba8b213f1d6cac7095dc5ccc3e5c" exitCode=0 Nov 03 21:05:13.238657 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:13.238455 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4" Nov 03 21:05:13.238657 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:13.238511 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4" event={"ID":"655cdf17-2be9-4e5d-bd3a-03b701349675","Type":"ContainerDied","Data":"a546c9919bad4fe0ed256ac0106000294e1bba8b213f1d6cac7095dc5ccc3e5c"} Nov 03 21:05:13.238657 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:13.238618 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4" event={"ID":"655cdf17-2be9-4e5d-bd3a-03b701349675","Type":"ContainerDied","Data":"ebf1012374541426c385e1094e9cc30e1dc948064bbe7a54848066747b4c9452"} Nov 03 21:05:13.238657 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:13.238649 4493 scope.go:122] "RemoveContainer" containerID="a546c9919bad4fe0ed256ac0106000294e1bba8b213f1d6cac7095dc5ccc3e5c" Nov 03 21:05:13.250206 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:13.250178 4493 scope.go:122] "RemoveContainer" containerID="a546c9919bad4fe0ed256ac0106000294e1bba8b213f1d6cac7095dc5ccc3e5c" Nov 03 21:05:13.250553 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:05:13.250524 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a546c9919bad4fe0ed256ac0106000294e1bba8b213f1d6cac7095dc5ccc3e5c\": container with ID starting with a546c9919bad4fe0ed256ac0106000294e1bba8b213f1d6cac7095dc5ccc3e5c not found: ID does not exist" containerID="a546c9919bad4fe0ed256ac0106000294e1bba8b213f1d6cac7095dc5ccc3e5c" Nov 03 21:05:13.250629 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:13.250566 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a546c9919bad4fe0ed256ac0106000294e1bba8b213f1d6cac7095dc5ccc3e5c"} err="failed to get container status \"a546c9919bad4fe0ed256ac0106000294e1bba8b213f1d6cac7095dc5ccc3e5c\": rpc error: code = NotFound desc = could not find container \"a546c9919bad4fe0ed256ac0106000294e1bba8b213f1d6cac7095dc5ccc3e5c\": container with ID starting with a546c9919bad4fe0ed256ac0106000294e1bba8b213f1d6cac7095dc5ccc3e5c not found: ID does not exist" Nov 03 21:05:13.260956 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:13.260877 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/655cdf17-2be9-4e5d-bd3a-03b701349675-metrics-certs\" (UniqueName: \"kubernetes.io/secret/655cdf17-2be9-4e5d-bd3a-03b701349675-metrics-certs\") pod \"655cdf17-2be9-4e5d-bd3a-03b701349675\" (UID: \"655cdf17-2be9-4e5d-bd3a-03b701349675\") " Nov 03 21:05:13.261203 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:13.261183 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/655cdf17-2be9-4e5d-bd3a-03b701349675-kube-api-access-wdk6m\" (UniqueName: \"kubernetes.io/projected/655cdf17-2be9-4e5d-bd3a-03b701349675-kube-api-access-wdk6m\") pod \"655cdf17-2be9-4e5d-bd3a-03b701349675\" (UID: \"655cdf17-2be9-4e5d-bd3a-03b701349675\") " Nov 03 21:05:13.261316 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:13.261293 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/655cdf17-2be9-4e5d-bd3a-03b701349675-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655cdf17-2be9-4e5d-bd3a-03b701349675-maas-api-tls\") pod \"655cdf17-2be9-4e5d-bd3a-03b701349675\" (UID: \"655cdf17-2be9-4e5d-bd3a-03b701349675\") " Nov 03 21:05:13.261398 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:13.261345 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/655cdf17-2be9-4e5d-bd3a-03b701349675-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/655cdf17-2be9-4e5d-bd3a-03b701349675-openshift-service-ca\") pod \"655cdf17-2be9-4e5d-bd3a-03b701349675\" (UID: \"655cdf17-2be9-4e5d-bd3a-03b701349675\") " Nov 03 21:05:13.261903 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:13.261876 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/655cdf17-2be9-4e5d-bd3a-03b701349675-openshift-service-ca" pod "655cdf17-2be9-4e5d-bd3a-03b701349675" (UID: "655cdf17-2be9-4e5d-bd3a-03b701349675"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:13.268427 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:13.266106 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655cdf17-2be9-4e5d-bd3a-03b701349675-kube-api-access-wdk6m" pod "655cdf17-2be9-4e5d-bd3a-03b701349675" (UID: "655cdf17-2be9-4e5d-bd3a-03b701349675"). InnerVolumeSpecName "kube-api-access-wdk6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:13.268427 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:13.266356 4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/655cdf17-2be9-4e5d-bd3a-03b701349675-openshift-service-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:13.268427 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:13.266377 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdk6m\" (UniqueName: \"kubernetes.io/projected/655cdf17-2be9-4e5d-bd3a-03b701349675-kube-api-access-wdk6m\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:13.268427 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:13.267726 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/655cdf17-2be9-4e5d-bd3a-03b701349675-metrics-certs" pod "655cdf17-2be9-4e5d-bd3a-03b701349675" (UID: "655cdf17-2be9-4e5d-bd3a-03b701349675"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:05:13.270976 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:13.270953 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/655cdf17-2be9-4e5d-bd3a-03b701349675-maas-api-tls" pod "655cdf17-2be9-4e5d-bd3a-03b701349675" (UID: "655cdf17-2be9-4e5d-bd3a-03b701349675"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:05:13.375712 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:13.374556 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655cdf17-2be9-4e5d-bd3a-03b701349675-maas-api-tls\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:13.375712 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:13.374591 4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655cdf17-2be9-4e5d-bd3a-03b701349675-metrics-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:13.653713 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:13.653626 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4"] Nov 03 21:05:13.700400 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:13.700368 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e6aaa6-6f9f68bbff-ljmg4"] Nov 03 21:05:14.563259 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:14.563211 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="655cdf17-2be9-4e5d-bd3a-03b701349675" path="/var/lib/kubelet/pods/655cdf17-2be9-4e5d-bd3a-03b701349675/volumes" Nov 03 21:05:21.746431 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:05:21.746392 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:05:24.410731 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:24.410689 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5"] Nov 03 21:05:24.411263 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:24.410907 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" podUID="1ad161f0-c9cb-4341-a7a8-55b24e1c1184" containerName="maas-api" containerID="cri-o://08d5558ead08ca292eeccf374a629f393c6df3458db9f5df9c2ba2b556d3524b" gracePeriod=30 Nov 03 21:05:24.476483 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:24.476208 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" podUID="1ad161f0-c9cb-4341-a7a8-55b24e1c1184" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.123:8443/health\": dial tcp 10.133.0.123:8443: connect: connection refused" Nov 03 21:05:24.762065 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:24.762040 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" Nov 03 21:05:24.912358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:24.910819 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-metrics-certs\") pod \"1ad161f0-c9cb-4341-a7a8-55b24e1c1184\" (UID: \"1ad161f0-c9cb-4341-a7a8-55b24e1c1184\") " Nov 03 21:05:24.912358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:24.910944 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-openshift-service-ca\") pod \"1ad161f0-c9cb-4341-a7a8-55b24e1c1184\" (UID: \"1ad161f0-c9cb-4341-a7a8-55b24e1c1184\") " Nov 03 21:05:24.912358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:24.911048 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-kube-api-access-gs2fl\" (UniqueName: \"kubernetes.io/projected/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-kube-api-access-gs2fl\") pod \"1ad161f0-c9cb-4341-a7a8-55b24e1c1184\" (UID: \"1ad161f0-c9cb-4341-a7a8-55b24e1c1184\") " Nov 03 21:05:24.912358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:24.911131 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-maas-api-tls\") pod \"1ad161f0-c9cb-4341-a7a8-55b24e1c1184\" (UID: \"1ad161f0-c9cb-4341-a7a8-55b24e1c1184\") " Nov 03 21:05:24.912743 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:24.912573 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-openshift-service-ca" pod "1ad161f0-c9cb-4341-a7a8-55b24e1c1184" (UID: "1ad161f0-c9cb-4341-a7a8-55b24e1c1184"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:24.916355 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:24.915199 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-maas-api-tls" pod "1ad161f0-c9cb-4341-a7a8-55b24e1c1184" (UID: "1ad161f0-c9cb-4341-a7a8-55b24e1c1184"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:05:24.916355 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:24.915905 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-kube-api-access-gs2fl" pod "1ad161f0-c9cb-4341-a7a8-55b24e1c1184" (UID: "1ad161f0-c9cb-4341-a7a8-55b24e1c1184"). InnerVolumeSpecName "kube-api-access-gs2fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:24.916651 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:24.916625 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-metrics-certs" pod "1ad161f0-c9cb-4341-a7a8-55b24e1c1184" (UID: "1ad161f0-c9cb-4341-a7a8-55b24e1c1184"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:05:25.030485 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:25.024719 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs2fl\" (UniqueName: \"kubernetes.io/projected/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-kube-api-access-gs2fl\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:25.030485 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:25.024758 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-maas-api-tls\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:25.030485 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:25.024774 4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-metrics-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:25.030485 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:25.024792 4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ad161f0-c9cb-4341-a7a8-55b24e1c1184-openshift-service-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:25.371496 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:25.371463 4493 generic.go:356] "Generic (PLEG): container finished" podID="1ad161f0-c9cb-4341-a7a8-55b24e1c1184" containerID="08d5558ead08ca292eeccf374a629f393c6df3458db9f5df9c2ba2b556d3524b" exitCode=0 Nov 03 21:05:25.371685 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:25.371510 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" Nov 03 21:05:25.371685 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:25.371538 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" event={"ID":"1ad161f0-c9cb-4341-a7a8-55b24e1c1184","Type":"ContainerDied","Data":"08d5558ead08ca292eeccf374a629f393c6df3458db9f5df9c2ba2b556d3524b"} Nov 03 21:05:25.371685 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:25.371590 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5" event={"ID":"1ad161f0-c9cb-4341-a7a8-55b24e1c1184","Type":"ContainerDied","Data":"17aefc2025642dfccc9ce8e7c45a13a2c638594ae39087920585dfc165d60337"} Nov 03 21:05:25.371685 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:25.371613 4493 scope.go:122] "RemoveContainer" containerID="08d5558ead08ca292eeccf374a629f393c6df3458db9f5df9c2ba2b556d3524b" Nov 03 21:05:25.383080 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:25.383061 4493 scope.go:122] "RemoveContainer" containerID="08d5558ead08ca292eeccf374a629f393c6df3458db9f5df9c2ba2b556d3524b" Nov 03 21:05:25.389206 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:05:25.383357 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d5558ead08ca292eeccf374a629f393c6df3458db9f5df9c2ba2b556d3524b\": container with ID starting with 08d5558ead08ca292eeccf374a629f393c6df3458db9f5df9c2ba2b556d3524b not found: ID does not exist" containerID="08d5558ead08ca292eeccf374a629f393c6df3458db9f5df9c2ba2b556d3524b" Nov 03 21:05:25.389206 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:25.383416 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d5558ead08ca292eeccf374a629f393c6df3458db9f5df9c2ba2b556d3524b"} err="failed to get container status \"08d5558ead08ca292eeccf374a629f393c6df3458db9f5df9c2ba2b556d3524b\": rpc error: code = NotFound desc = could not find container \"08d5558ead08ca292eeccf374a629f393c6df3458db9f5df9c2ba2b556d3524b\": container with ID starting with 08d5558ead08ca292eeccf374a629f393c6df3458db9f5df9c2ba2b556d3524b not found: ID does not exist" Nov 03 21:05:25.521368 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:25.510189 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6"] Nov 03 21:05:25.582218 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:25.580761 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" podUID="e51960f4-5af5-4d6b-ae00-29b0e54c5ba3" containerName="istio-proxy" containerID="cri-o://59b490cd40499afb454f31bb49f2f64202e6939b66f858746c7e1d090709fcd8" gracePeriod=30 Nov 03 21:05:25.765422 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:25.748419 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5"] Nov 03 21:05:25.769905 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:25.769010 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-80cbc0-6d97bd8858-mfcg5"] Nov 03 21:05:26.562643 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:26.562612 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ad161f0-c9cb-4341-a7a8-55b24e1c1184" path="/var/lib/kubelet/pods/1ad161f0-c9cb-4341-a7a8-55b24e1c1184/volumes" Nov 03 21:05:27.314876 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:27.314838 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k"] Nov 03 21:05:27.315135 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:27.315099 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" podUID="7bae91ea-007d-4af0-861d-9e7fc6610997" containerName="istio-proxy" containerID="cri-o://f075e686dcae4c2b4dc3809237934e3930079f7b9a97b053dc038bfd09915d5f" gracePeriod=30 Nov 03 21:05:30.914259 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.914196 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 21:05:30.981537 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.981470 4493 patch_prober.go:29] interesting pod/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:05:30.981737 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.981563 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" podUID="21488ac9-bfba-4bb5-ba76-8c76c7c9fa21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:05:30.983699 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.983666 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-podinfo\") pod \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " Nov 03 21:05:30.983847 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.983763 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istiod-ca-cert\") pod \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " Nov 03 21:05:30.983847 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.983817 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-data\") pod \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " Nov 03 21:05:30.983948 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.983916 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-workload-socket\") pod \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " Nov 03 21:05:30.984234 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.984054 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-envoy\") pod \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " Nov 03 21:05:30.984234 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.984102 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-workload-certs\") pod \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " Nov 03 21:05:30.984234 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.984153 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-token\" (UniqueName: \"kubernetes.io/projected/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-token\") pod \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " Nov 03 21:05:30.984536 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.984241 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-credential-socket\") pod \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " Nov 03 21:05:30.984536 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.984274 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-workload-socket" pod "e51960f4-5af5-4d6b-ae00-29b0e54c5ba3" (UID: "e51960f4-5af5-4d6b-ae00-29b0e54c5ba3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:30.984536 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.984303 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-kube-api-access-nx2dz\" (UniqueName: \"kubernetes.io/projected/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-kube-api-access-nx2dz\") pod \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\" (UID: \"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3\") " Nov 03 21:05:30.986376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.984808 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istiod-ca-cert" pod "e51960f4-5af5-4d6b-ae00-29b0e54c5ba3" (UID: "e51960f4-5af5-4d6b-ae00-29b0e54c5ba3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:30.986517 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.986485 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-data" pod "e51960f4-5af5-4d6b-ae00-29b0e54c5ba3" (UID: "e51960f4-5af5-4d6b-ae00-29b0e54c5ba3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:30.987775 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.987735 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-workload-certs" pod "e51960f4-5af5-4d6b-ae00-29b0e54c5ba3" (UID: "e51960f4-5af5-4d6b-ae00-29b0e54c5ba3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:30.988054 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.988021 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-credential-socket" pod "e51960f4-5af5-4d6b-ae00-29b0e54c5ba3" (UID: "e51960f4-5af5-4d6b-ae00-29b0e54c5ba3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:30.988912 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.988889 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-podinfo" pod "e51960f4-5af5-4d6b-ae00-29b0e54c5ba3" (UID: "e51960f4-5af5-4d6b-ae00-29b0e54c5ba3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:05:30.991023 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.990996 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:30.991163 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.991030 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:30.991163 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.991044 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:30.991163 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.991059 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:30.991163 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.991072 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:30.991163 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.991085 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:30.991987 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.991950 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-envoy" pod "e51960f4-5af5-4d6b-ae00-29b0e54c5ba3" (UID: "e51960f4-5af5-4d6b-ae00-29b0e54c5ba3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:30.992319 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.992285 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-token" pod "e51960f4-5af5-4d6b-ae00-29b0e54c5ba3" (UID: "e51960f4-5af5-4d6b-ae00-29b0e54c5ba3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:30.994742 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:30.994717 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-kube-api-access-nx2dz" pod "e51960f4-5af5-4d6b-ae00-29b0e54c5ba3" (UID: "e51960f4-5af5-4d6b-ae00-29b0e54c5ba3"). InnerVolumeSpecName "kube-api-access-nx2dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:31.092436 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:31.092396 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:31.092436 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:31.092433 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:31.092729 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:31.092450 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx2dz\" (UniqueName: \"kubernetes.io/projected/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3-kube-api-access-nx2dz\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:31.456864 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:31.456776 4493 generic.go:356] "Generic (PLEG): container finished" podID="21488ac9-bfba-4bb5-ba76-8c76c7c9fa21" containerID="2ebfbf22f684f438ec47b8de4b01b53a8c35ff0ebe31c09ab257425d29f92056" exitCode=0 Nov 03 21:05:31.456864 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:31.456846 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" event={"ID":"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21","Type":"ContainerDied","Data":"2ebfbf22f684f438ec47b8de4b01b53a8c35ff0ebe31c09ab257425d29f92056"} Nov 03 21:05:31.457425 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:31.457398 4493 scope.go:122] "RemoveContainer" containerID="2ebfbf22f684f438ec47b8de4b01b53a8c35ff0ebe31c09ab257425d29f92056" Nov 03 21:05:31.459249 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:31.459224 4493 generic.go:356] "Generic (PLEG): container finished" podID="e51960f4-5af5-4d6b-ae00-29b0e54c5ba3" containerID="59b490cd40499afb454f31bb49f2f64202e6939b66f858746c7e1d090709fcd8" exitCode=0 Nov 03 21:05:31.459392 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:31.459278 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" Nov 03 21:05:31.459392 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:31.459307 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" event={"ID":"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3","Type":"ContainerDied","Data":"59b490cd40499afb454f31bb49f2f64202e6939b66f858746c7e1d090709fcd8"} Nov 03 21:05:31.459392 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:31.459379 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6" event={"ID":"e51960f4-5af5-4d6b-ae00-29b0e54c5ba3","Type":"ContainerDied","Data":"3538c56d951d3b25b0b4c4c1fe4a67fc1f861d983cee4653d8fe6170561bf113"} Nov 03 21:05:31.459583 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:31.459413 4493 scope.go:122] "RemoveContainer" containerID="59b490cd40499afb454f31bb49f2f64202e6939b66f858746c7e1d090709fcd8" Nov 03 21:05:31.459935 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:31.459898 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:05:31.460263 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:31.460237 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:05:31.460583 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:31.460557 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:05:31.488511 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:31.488488 4493 scope.go:122] "RemoveContainer" containerID="c566abe6eaaae1012a09bfbac4215eaed8a252ee2323151f7334bcb5c595faaf" Nov 03 21:05:31.498889 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:31.498870 4493 scope.go:122] "RemoveContainer" containerID="59b490cd40499afb454f31bb49f2f64202e6939b66f858746c7e1d090709fcd8" Nov 03 21:05:31.499202 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:05:31.499176 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59b490cd40499afb454f31bb49f2f64202e6939b66f858746c7e1d090709fcd8\": container with ID starting with 59b490cd40499afb454f31bb49f2f64202e6939b66f858746c7e1d090709fcd8 not found: ID does not exist" containerID="59b490cd40499afb454f31bb49f2f64202e6939b66f858746c7e1d090709fcd8" Nov 03 21:05:31.499288 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:31.499213 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59b490cd40499afb454f31bb49f2f64202e6939b66f858746c7e1d090709fcd8"} err="failed to get container status \"59b490cd40499afb454f31bb49f2f64202e6939b66f858746c7e1d090709fcd8\": rpc error: code = NotFound desc = could not find container \"59b490cd40499afb454f31bb49f2f64202e6939b66f858746c7e1d090709fcd8\": container with ID starting with 59b490cd40499afb454f31bb49f2f64202e6939b66f858746c7e1d090709fcd8 not found: ID does not exist" Nov 03 21:05:31.499288 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:31.499239 4493 scope.go:122] "RemoveContainer" containerID="c566abe6eaaae1012a09bfbac4215eaed8a252ee2323151f7334bcb5c595faaf" Nov 03 21:05:31.499585 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:05:31.499566 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c566abe6eaaae1012a09bfbac4215eaed8a252ee2323151f7334bcb5c595faaf\": container with ID starting with c566abe6eaaae1012a09bfbac4215eaed8a252ee2323151f7334bcb5c595faaf not found: ID does not exist" containerID="c566abe6eaaae1012a09bfbac4215eaed8a252ee2323151f7334bcb5c595faaf" Nov 03 21:05:31.499660 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:31.499591 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c566abe6eaaae1012a09bfbac4215eaed8a252ee2323151f7334bcb5c595faaf"} err="failed to get container status \"c566abe6eaaae1012a09bfbac4215eaed8a252ee2323151f7334bcb5c595faaf\": rpc error: code = NotFound desc = could not find container \"c566abe6eaaae1012a09bfbac4215eaed8a252ee2323151f7334bcb5c595faaf\": container with ID starting with c566abe6eaaae1012a09bfbac4215eaed8a252ee2323151f7334bcb5c595faaf not found: ID does not exist" Nov 03 21:05:31.673395 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:31.673314 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6"] Nov 03 21:05:31.676583 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:31.676559 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-1ec1b6-openshift-default-7cf97bf64c-mbgz6"] Nov 03 21:05:32.470018 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.469987 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" event={"ID":"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21","Type":"ContainerStarted","Data":"0ced931349b00e04af2fef095509d00e698fdefb6afeeec03e58a1a2f0cd71a4"} Nov 03 21:05:32.471893 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.471867 4493 generic.go:356] "Generic (PLEG): container finished" podID="7bae91ea-007d-4af0-861d-9e7fc6610997" containerID="f075e686dcae4c2b4dc3809237934e3930079f7b9a97b053dc038bfd09915d5f" exitCode=0 Nov 03 21:05:32.472036 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.471941 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" event={"ID":"7bae91ea-007d-4af0-861d-9e7fc6610997","Type":"ContainerDied","Data":"f075e686dcae4c2b4dc3809237934e3930079f7b9a97b053dc038bfd09915d5f"} Nov 03 21:05:32.563799 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.563767 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e51960f4-5af5-4d6b-ae00-29b0e54c5ba3" path="/var/lib/kubelet/pods/e51960f4-5af5-4d6b-ae00-29b0e54c5ba3/volumes" Nov 03 21:05:32.639070 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.639048 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 21:05:32.718413 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.718373 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-workload-certs\") pod \"7bae91ea-007d-4af0-861d-9e7fc6610997\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " Nov 03 21:05:32.718612 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.718506 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-workload-socket\") pod \"7bae91ea-007d-4af0-861d-9e7fc6610997\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " Nov 03 21:05:32.718612 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.718534 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-credential-socket\") pod \"7bae91ea-007d-4af0-861d-9e7fc6610997\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " Nov 03 21:05:32.718612 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.718584 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bae91ea-007d-4af0-861d-9e7fc6610997-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bae91ea-007d-4af0-861d-9e7fc6610997-istiod-ca-cert\") pod \"7bae91ea-007d-4af0-861d-9e7fc6610997\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " Nov 03 21:05:32.718790 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.718653 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bae91ea-007d-4af0-861d-9e7fc6610997-kube-api-access-n7979\" (UniqueName: \"kubernetes.io/projected/7bae91ea-007d-4af0-861d-9e7fc6610997-kube-api-access-n7979\") pod \"7bae91ea-007d-4af0-861d-9e7fc6610997\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " Nov 03 21:05:32.718790 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.718679 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-workload-certs" pod "7bae91ea-007d-4af0-861d-9e7fc6610997" (UID: "7bae91ea-007d-4af0-861d-9e7fc6610997"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:32.718790 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.718702 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-data\") pod \"7bae91ea-007d-4af0-861d-9e7fc6610997\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " Nov 03 21:05:32.718790 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.718767 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-token\" (UniqueName: \"kubernetes.io/projected/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-token\") pod \"7bae91ea-007d-4af0-861d-9e7fc6610997\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " Nov 03 21:05:32.718989 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.718809 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-workload-socket" pod "7bae91ea-007d-4af0-861d-9e7fc6610997" (UID: "7bae91ea-007d-4af0-861d-9e7fc6610997"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:32.718989 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.718825 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-credential-socket" pod "7bae91ea-007d-4af0-861d-9e7fc6610997" (UID: "7bae91ea-007d-4af0-861d-9e7fc6610997"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:32.718989 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.718851 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-envoy\") pod \"7bae91ea-007d-4af0-861d-9e7fc6610997\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " Nov 03 21:05:32.718989 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.718889 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-podinfo\") pod \"7bae91ea-007d-4af0-861d-9e7fc6610997\" (UID: \"7bae91ea-007d-4af0-861d-9e7fc6610997\") " Nov 03 21:05:32.718989 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.718971 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-data" pod "7bae91ea-007d-4af0-861d-9e7fc6610997" (UID: "7bae91ea-007d-4af0-861d-9e7fc6610997"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:32.719253 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.719011 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bae91ea-007d-4af0-861d-9e7fc6610997-istiod-ca-cert" pod "7bae91ea-007d-4af0-861d-9e7fc6610997" (UID: "7bae91ea-007d-4af0-861d-9e7fc6610997"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:32.719852 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.719822 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:32.720005 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.719857 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:32.720005 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.719876 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:32.720005 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.719893 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:32.720005 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.719908 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bae91ea-007d-4af0-861d-9e7fc6610997-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:32.721243 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.721217 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-envoy" pod "7bae91ea-007d-4af0-861d-9e7fc6610997" (UID: "7bae91ea-007d-4af0-861d-9e7fc6610997"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:32.721355 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.721297 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bae91ea-007d-4af0-861d-9e7fc6610997-kube-api-access-n7979" pod "7bae91ea-007d-4af0-861d-9e7fc6610997" (UID: "7bae91ea-007d-4af0-861d-9e7fc6610997"). InnerVolumeSpecName "kube-api-access-n7979". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:32.721355 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.721318 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-token" pod "7bae91ea-007d-4af0-861d-9e7fc6610997" (UID: "7bae91ea-007d-4af0-861d-9e7fc6610997"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:32.721716 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.721702 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-podinfo" pod "7bae91ea-007d-4af0-861d-9e7fc6610997" (UID: "7bae91ea-007d-4af0-861d-9e7fc6610997"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:05:32.822754 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.821784 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7979\" (UniqueName: \"kubernetes.io/projected/7bae91ea-007d-4af0-861d-9e7fc6610997-kube-api-access-n7979\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:32.822754 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.821815 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:32.822754 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.821826 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:32.822754 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:32.821835 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bae91ea-007d-4af0-861d-9e7fc6610997-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:33.258767 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:33.258721 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm"] Nov 03 21:05:33.258980 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:33.258954 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" podUID="2c81f07e-ff42-4534-856a-01f0069a8cb2" containerName="istio-proxy" containerID="cri-o://a366ad942960a4fbe767b18240df00480315a431233f8e84c74ad73f79bd837c" gracePeriod=30 Nov 03 21:05:33.314439 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:33.309477 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:05:33.314439 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:33.309603 4493 patch_prober.go:29] interesting pod/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:05:33.314439 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:33.309648 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" podUID="21488ac9-bfba-4bb5-ba76-8c76c7c9fa21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 21:05:33.540039 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:33.539960 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" Nov 03 21:05:33.540649 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:33.540450 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k" event={"ID":"7bae91ea-007d-4af0-861d-9e7fc6610997","Type":"ContainerDied","Data":"328d1a247a508232a1119f6f17071c6750886db5dae5c0fe95f85ef46fa1728a"} Nov 03 21:05:33.540649 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:33.540516 4493 scope.go:122] "RemoveContainer" containerID="f075e686dcae4c2b4dc3809237934e3930079f7b9a97b053dc038bfd09915d5f" Nov 03 21:05:33.713323 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:33.713288 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k"] Nov 03 21:05:33.716575 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:33.716553 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-s4r2k"] Nov 03 21:05:34.305493 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.303690 4493 patch_prober.go:29] interesting pod/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:05:34.305493 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.303759 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" podUID="21488ac9-bfba-4bb5-ba76-8c76c7c9fa21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.464307 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs"] Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465205 4493 cpu_manager.go:414] "containerMap: removing container" podUID="87ddbb89-1468-4f64-9250-3a2b32ecdfb3" containerName="payload-pre-processing" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465226 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87ddbb89-1468-4f64-9250-3a2b32ecdfb3" containerName="payload-pre-processing" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465247 4493 cpu_manager.go:414] "containerMap: removing container" podUID="6a0808b2-7798-4f60-9435-063b72bbdb21" containerName="maas-api" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465255 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a0808b2-7798-4f60-9435-063b72bbdb21" containerName="maas-api" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465278 4493 cpu_manager.go:414] "containerMap: removing container" podUID="33d6f310-96b9-4029-8f11-179458b5b17c" containerName="payload-processing" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465288 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33d6f310-96b9-4029-8f11-179458b5b17c" containerName="payload-processing" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465301 4493 cpu_manager.go:414] "containerMap: removing container" podUID="d12d909a-e17b-470b-9ba6-164d1dbffba1" containerName="payload-pre-processing" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465308 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d12d909a-e17b-470b-9ba6-164d1dbffba1" containerName="payload-pre-processing" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465327 4493 cpu_manager.go:414] "containerMap: removing container" podUID="1ad161f0-c9cb-4341-a7a8-55b24e1c1184" containerName="maas-api" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465352 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ad161f0-c9cb-4341-a7a8-55b24e1c1184" containerName="maas-api" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465370 4493 cpu_manager.go:414] "containerMap: removing container" podUID="f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898" containerName="payload-processing" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465375 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898" containerName="payload-processing" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465386 4493 cpu_manager.go:414] "containerMap: removing container" podUID="7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac" containerName="payload-pre-processing" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465391 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac" containerName="payload-pre-processing" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465399 4493 cpu_manager.go:414] "containerMap: removing container" podUID="7bae91ea-007d-4af0-861d-9e7fc6610997" containerName="istio-proxy" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465404 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bae91ea-007d-4af0-861d-9e7fc6610997" containerName="istio-proxy" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465410 4493 cpu_manager.go:414] "containerMap: removing container" podUID="655cdf17-2be9-4e5d-bd3a-03b701349675" containerName="maas-api" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465418 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="655cdf17-2be9-4e5d-bd3a-03b701349675" containerName="maas-api" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465435 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e51960f4-5af5-4d6b-ae00-29b0e54c5ba3" containerName="istio-proxy" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465442 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e51960f4-5af5-4d6b-ae00-29b0e54c5ba3" containerName="istio-proxy" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465452 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e51960f4-5af5-4d6b-ae00-29b0e54c5ba3" containerName="istio-proxy" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465459 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e51960f4-5af5-4d6b-ae00-29b0e54c5ba3" containerName="istio-proxy" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465483 4493 cpu_manager.go:414] "containerMap: removing container" podUID="40b21981-a7ed-4e7e-8ac0-23ac2927547d" containerName="payload-processing" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465490 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b21981-a7ed-4e7e-8ac0-23ac2927547d" containerName="payload-processing" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465509 4493 cpu_manager.go:414] "containerMap: removing container" podUID="2521a657-32ee-4d9b-a1e6-a319fd952387" containerName="payload-pre-processing" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465526 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2521a657-32ee-4d9b-a1e6-a319fd952387" containerName="payload-pre-processing" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465704 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ad161f0-c9cb-4341-a7a8-55b24e1c1184" containerName="maas-api" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465726 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9bf9f02-cd0d-4f3a-8aaa-d83cbd778898" containerName="payload-processing" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465741 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ac6a60c-6cb3-4c6c-9e62-2e9732f2f8ac" containerName="payload-pre-processing" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465757 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bae91ea-007d-4af0-861d-9e7fc6610997" containerName="istio-proxy" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465771 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="655cdf17-2be9-4e5d-bd3a-03b701349675" containerName="maas-api" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465787 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e51960f4-5af5-4d6b-ae00-29b0e54c5ba3" containerName="istio-proxy" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465802 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e51960f4-5af5-4d6b-ae00-29b0e54c5ba3" containerName="istio-proxy" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465815 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b21981-a7ed-4e7e-8ac0-23ac2927547d" containerName="payload-processing" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465824 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="2521a657-32ee-4d9b-a1e6-a319fd952387" containerName="payload-pre-processing" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465836 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="87ddbb89-1468-4f64-9250-3a2b32ecdfb3" containerName="payload-pre-processing" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465845 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a0808b2-7798-4f60-9435-063b72bbdb21" containerName="maas-api" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465856 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="33d6f310-96b9-4029-8f11-179458b5b17c" containerName="payload-processing" Nov 03 21:05:34.468376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.465862 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d12d909a-e17b-470b-9ba6-164d1dbffba1" containerName="payload-pre-processing" Nov 03 21:05:34.471216 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.471194 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.547241 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.540630 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c14420-openshift-default-dockercfg-zdwcp\"" Nov 03 21:05:34.547241 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.546094 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs"] Nov 03 21:05:34.564069 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.563981 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bae91ea-007d-4af0-861d-9e7fc6610997" path="/var/lib/kubelet/pods/7bae91ea-007d-4af0-861d-9e7fc6610997/volumes" Nov 03 21:05:34.570663 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.570629 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b5e9e0-94df-40e7-b031-15c11babc655-istiod-ca-cert\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.570820 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.570699 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-envoy\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.570820 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.570772 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-data\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.570820 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.570808 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-credential-socket\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.570979 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.570873 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-podinfo\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.570979 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.570909 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-workload-socket\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.571077 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.570981 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-workload-certs\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.571135 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.571118 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-token\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.571189 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.571179 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wfx2\" (UniqueName: \"kubernetes.io/projected/b5b5e9e0-94df-40e7-b031-15c11babc655-kube-api-access-2wfx2\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.678358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.674734 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-workload-certs\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.678358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.674929 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-token\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.678358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.674987 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wfx2\" (UniqueName: \"kubernetes.io/projected/b5b5e9e0-94df-40e7-b031-15c11babc655-kube-api-access-2wfx2\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.678358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.675081 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b5e9e0-94df-40e7-b031-15c11babc655-istiod-ca-cert\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.678358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.675123 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-envoy\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.678358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.675201 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-data\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.678358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.675232 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-credential-socket\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.678358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.675301 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-podinfo\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.678358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.675353 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-workload-socket\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.678358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.676104 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-workload-certs\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.683796 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.683764 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-data\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.697424 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.684763 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b5e9e0-94df-40e7-b031-15c11babc655-istiod-ca-cert\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.697680 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.696755 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-workload-socket\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.697815 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.696976 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-credential-socket\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.731844 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.730991 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-podinfo\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.733466 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.733437 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-envoy\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.741362 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.736645 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wfx2\" (UniqueName: \"kubernetes.io/projected/b5b5e9e0-94df-40e7-b031-15c11babc655-kube-api-access-2wfx2\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.751658 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.751352 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-token\") pod \"e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:34.791360 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:34.787569 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:35.233602 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:35.233530 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs"] Nov 03 21:05:35.277221 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:05:35.277187 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b5e9e0_94df_40e7_b031_15c11babc655.slice/crio-6730834c1a60dd1d7de0864ea8756bca7247754dc26cd4713e7b3c233d5f97f6 WatchSource:0}: Error finding container 6730834c1a60dd1d7de0864ea8756bca7247754dc26cd4713e7b3c233d5f97f6: Status 404 returned error can't find the container with id 6730834c1a60dd1d7de0864ea8756bca7247754dc26cd4713e7b3c233d5f97f6 Nov 03 21:05:35.280051 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:35.280019 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:05:35.280140 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:35.280089 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:05:35.280140 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:35.280116 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:05:35.308813 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:35.303693 4493 patch_prober.go:29] interesting pod/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:05:35.308813 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:35.303743 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" podUID="21488ac9-bfba-4bb5-ba76-8c76c7c9fa21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 21:05:35.558226 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:35.558196 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" event={"ID":"b5b5e9e0-94df-40e7-b031-15c11babc655","Type":"ContainerStarted","Data":"ff8f9c1a71bb1c053be98b46bc6bb774a160468595af4f4fed0660ea97293553"} Nov 03 21:05:35.558683 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:35.558244 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" event={"ID":"b5b5e9e0-94df-40e7-b031-15c11babc655","Type":"ContainerStarted","Data":"6730834c1a60dd1d7de0864ea8756bca7247754dc26cd4713e7b3c233d5f97f6"} Nov 03 21:05:35.671073 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:35.671023 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" podStartSLOduration=1.6710057360000001 podStartE2EDuration="1.671005736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:35.669855671 +0000 UTC m=+2207.227846211" watchObservedRunningTime="2026-11-03 21:05:35.671005736 +0000 UTC m=+2207.228996235" Nov 03 21:05:35.806360 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:35.805383 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:36.301694 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:36.301655 4493 patch_prober.go:29] interesting pod/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:05:36.301999 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:36.301973 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" podUID="21488ac9-bfba-4bb5-ba76-8c76c7c9fa21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 21:05:36.813037 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:36.813005 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:37.309408 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:37.302434 4493 patch_prober.go:29] interesting pod/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:05:37.309408 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:37.302492 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" podUID="21488ac9-bfba-4bb5-ba76-8c76c7c9fa21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 21:05:37.576762 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:37.576650 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:37.578202 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:37.577998 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:05:38.304566 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.304542 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:05:38.305060 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.305040 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:05:38.305811 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.305794 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:05:38.596108 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.596076 4493 generic.go:356] "Generic (PLEG): container finished" podID="2c81f07e-ff42-4534-856a-01f0069a8cb2" containerID="a366ad942960a4fbe767b18240df00480315a431233f8e84c74ad73f79bd837c" exitCode=0 Nov 03 21:05:38.596259 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.596151 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" event={"ID":"2c81f07e-ff42-4534-856a-01f0069a8cb2","Type":"ContainerDied","Data":"a366ad942960a4fbe767b18240df00480315a431233f8e84c74ad73f79bd837c"} Nov 03 21:05:38.596259 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.596214 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" event={"ID":"2c81f07e-ff42-4534-856a-01f0069a8cb2","Type":"ContainerDied","Data":"66b07a0a2a4b1185052ed70568c5222d9263b3c8ea06af18c78ddd51c0e8262a"} Nov 03 21:05:38.596259 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.596238 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66b07a0a2a4b1185052ed70568c5222d9263b3c8ea06af18c78ddd51c0e8262a" Nov 03 21:05:38.596259 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.596252 4493 scope.go:122] "RemoveContainer" containerID="61ea581538c24a83c09a29ac1953b83ee2007762b6e3f0cd1167d00aa0ed7b59" Nov 03 21:05:38.606300 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.606275 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:05:38.693133 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.693101 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg"] Nov 03 21:05:38.791512 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.791484 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c81f07e-ff42-4534-856a-01f0069a8cb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c81f07e-ff42-4534-856a-01f0069a8cb2-istiod-ca-cert\") pod \"2c81f07e-ff42-4534-856a-01f0069a8cb2\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " Nov 03 21:05:38.791716 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.791558 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c81f07e-ff42-4534-856a-01f0069a8cb2-kube-api-access-p9lcq\" (UniqueName: \"kubernetes.io/projected/2c81f07e-ff42-4534-856a-01f0069a8cb2-kube-api-access-p9lcq\") pod \"2c81f07e-ff42-4534-856a-01f0069a8cb2\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " Nov 03 21:05:38.791716 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.791583 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-workload-socket\") pod \"2c81f07e-ff42-4534-856a-01f0069a8cb2\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " Nov 03 21:05:38.791716 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.791611 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-workload-certs\") pod \"2c81f07e-ff42-4534-856a-01f0069a8cb2\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " Nov 03 21:05:38.791897 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.791722 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-credential-socket\") pod \"2c81f07e-ff42-4534-856a-01f0069a8cb2\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " Nov 03 21:05:38.791897 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.791790 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-token\" (UniqueName: \"kubernetes.io/projected/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-token\") pod \"2c81f07e-ff42-4534-856a-01f0069a8cb2\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " Nov 03 21:05:38.791897 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.791857 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-podinfo\") pod \"2c81f07e-ff42-4534-856a-01f0069a8cb2\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " Nov 03 21:05:38.792054 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.791944 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-envoy\") pod \"2c81f07e-ff42-4534-856a-01f0069a8cb2\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " Nov 03 21:05:38.792054 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.791984 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c81f07e-ff42-4534-856a-01f0069a8cb2-istiod-ca-cert" pod "2c81f07e-ff42-4534-856a-01f0069a8cb2" (UID: "2c81f07e-ff42-4534-856a-01f0069a8cb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:38.792054 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.792035 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-data\") pod \"2c81f07e-ff42-4534-856a-01f0069a8cb2\" (UID: \"2c81f07e-ff42-4534-856a-01f0069a8cb2\") " Nov 03 21:05:38.792224 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.792203 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-workload-certs" pod "2c81f07e-ff42-4534-856a-01f0069a8cb2" (UID: "2c81f07e-ff42-4534-856a-01f0069a8cb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:38.792485 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.792429 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-data" pod "2c81f07e-ff42-4534-856a-01f0069a8cb2" (UID: "2c81f07e-ff42-4534-856a-01f0069a8cb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:38.792713 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.792689 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-credential-socket" pod "2c81f07e-ff42-4534-856a-01f0069a8cb2" (UID: "2c81f07e-ff42-4534-856a-01f0069a8cb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:38.794564 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.793440 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-workload-socket" pod "2c81f07e-ff42-4534-856a-01f0069a8cb2" (UID: "2c81f07e-ff42-4534-856a-01f0069a8cb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:38.794564 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.794176 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:38.794564 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.794196 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c81f07e-ff42-4534-856a-01f0069a8cb2-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:38.794564 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.794211 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:38.794564 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.794224 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:38.794564 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.794235 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:38.795301 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.795278 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt"] Nov 03 21:05:38.795660 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.795631 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c81f07e-ff42-4534-856a-01f0069a8cb2-kube-api-access-p9lcq" pod "2c81f07e-ff42-4534-856a-01f0069a8cb2" (UID: "2c81f07e-ff42-4534-856a-01f0069a8cb2"). InnerVolumeSpecName "kube-api-access-p9lcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:38.796055 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.796035 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-token" pod "2c81f07e-ff42-4534-856a-01f0069a8cb2" (UID: "2c81f07e-ff42-4534-856a-01f0069a8cb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:38.797773 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.797749 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-envoy" pod "2c81f07e-ff42-4534-856a-01f0069a8cb2" (UID: "2c81f07e-ff42-4534-856a-01f0069a8cb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:38.798082 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.798055 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-podinfo" pod "2c81f07e-ff42-4534-856a-01f0069a8cb2" (UID: "2c81f07e-ff42-4534-856a-01f0069a8cb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:05:38.895179 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.895084 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9lcq\" (UniqueName: \"kubernetes.io/projected/2c81f07e-ff42-4534-856a-01f0069a8cb2-kube-api-access-p9lcq\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:38.895179 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.895117 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:38.895179 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.895132 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:38.895179 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:38.895145 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c81f07e-ff42-4534-856a-01f0069a8cb2-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:39.619470 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:39.619418 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm" Nov 03 21:05:39.844030 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:39.844001 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm"] Nov 03 21:05:39.851601 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:39.851572 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-80cbc0-openshift-default-85b7b8fc47-x6mwm"] Nov 03 21:05:39.879529 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:39.879441 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw"] Nov 03 21:05:39.879776 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:39.879744 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" podUID="08b80b2f-d192-42d5-9657-4d5084cdb967" containerName="istio-proxy" containerID="cri-o://80b183788d53186258439e88bc08ea093f0cd66f76940109eb7fe606692f3e45" gracePeriod=30 Nov 03 21:05:39.887553 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:39.886485 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz"] Nov 03 21:05:39.887553 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:39.886782 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" podUID="a50088fd-39b0-4118-8f90-c8e6b7fb3dd0" containerName="istio-proxy" containerID="cri-o://bfad999a5615d928e6071fddaabe2b24a682bc68d423453653c33f8110c60a3e" gracePeriod=30 Nov 03 21:05:40.572854 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:40.568024 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c81f07e-ff42-4534-856a-01f0069a8cb2" path="/var/lib/kubelet/pods/2c81f07e-ff42-4534-856a-01f0069a8cb2/volumes" Nov 03 21:05:43.693459 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:43.693402 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg" podUID="74a3fc99-772f-4628-b9bc-7754ae9e4b1f" containerName="payload-processing" containerID="cri-o://a5fd2c8d623e2a31bfb5b13e8bac667dd83d8e9a709f3bc7520209d28c6bf4e7" gracePeriod=25 Nov 03 21:05:43.796572 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:43.796514 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt" podUID="0c60698d-6705-4cfc-8a08-4246536566b5" containerName="payload-pre-processing" containerID="cri-o://e9812bd244502fd1b362287941ea2d4286cf420d65e38d5b75bb217877484f81" gracePeriod=25 Nov 03 21:05:43.986890 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:43.986861 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg" Nov 03 21:05:44.050807 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.050777 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a3fc99-772f-4628-b9bc-7754ae9e4b1f-kube-api-access-rkf9h\" (UniqueName: \"kubernetes.io/projected/74a3fc99-772f-4628-b9bc-7754ae9e4b1f-kube-api-access-rkf9h\") pod \"74a3fc99-772f-4628-b9bc-7754ae9e4b1f\" (UID: \"74a3fc99-772f-4628-b9bc-7754ae9e4b1f\") " Nov 03 21:05:44.051026 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.050922 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a3fc99-772f-4628-b9bc-7754ae9e4b1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a3fc99-772f-4628-b9bc-7754ae9e4b1f-plugins-config-volume\") pod \"74a3fc99-772f-4628-b9bc-7754ae9e4b1f\" (UID: \"74a3fc99-772f-4628-b9bc-7754ae9e4b1f\") " Nov 03 21:05:44.051836 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.051811 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a3fc99-772f-4628-b9bc-7754ae9e4b1f-plugins-config-volume" pod "74a3fc99-772f-4628-b9bc-7754ae9e4b1f" (UID: "74a3fc99-772f-4628-b9bc-7754ae9e4b1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:44.054268 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.054238 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a3fc99-772f-4628-b9bc-7754ae9e4b1f-kube-api-access-rkf9h" pod "74a3fc99-772f-4628-b9bc-7754ae9e4b1f" (UID: "74a3fc99-772f-4628-b9bc-7754ae9e4b1f"). InnerVolumeSpecName "kube-api-access-rkf9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:44.060735 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.060299 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt" Nov 03 21:05:44.152092 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.152058 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c60698d-6705-4cfc-8a08-4246536566b5-kube-api-access-mf9dd\" (UniqueName: \"kubernetes.io/projected/0c60698d-6705-4cfc-8a08-4246536566b5-kube-api-access-mf9dd\") pod \"0c60698d-6705-4cfc-8a08-4246536566b5\" (UID: \"0c60698d-6705-4cfc-8a08-4246536566b5\") " Nov 03 21:05:44.152275 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.152196 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c60698d-6705-4cfc-8a08-4246536566b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c60698d-6705-4cfc-8a08-4246536566b5-plugins-config-volume\") pod \"0c60698d-6705-4cfc-8a08-4246536566b5\" (UID: \"0c60698d-6705-4cfc-8a08-4246536566b5\") " Nov 03 21:05:44.156311 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.152617 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c60698d-6705-4cfc-8a08-4246536566b5-plugins-config-volume" pod "0c60698d-6705-4cfc-8a08-4246536566b5" (UID: "0c60698d-6705-4cfc-8a08-4246536566b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:44.156311 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.152996 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkf9h\" (UniqueName: \"kubernetes.io/projected/74a3fc99-772f-4628-b9bc-7754ae9e4b1f-kube-api-access-rkf9h\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:44.156311 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.153018 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a3fc99-772f-4628-b9bc-7754ae9e4b1f-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:44.156311 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.153032 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c60698d-6705-4cfc-8a08-4246536566b5-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:44.156624 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.156477 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c60698d-6705-4cfc-8a08-4246536566b5-kube-api-access-mf9dd" pod "0c60698d-6705-4cfc-8a08-4246536566b5" (UID: "0c60698d-6705-4cfc-8a08-4246536566b5"). InnerVolumeSpecName "kube-api-access-mf9dd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:44.253973 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.253888 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf9dd\" (UniqueName: \"kubernetes.io/projected/0c60698d-6705-4cfc-8a08-4246536566b5-kube-api-access-mf9dd\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:44.656561 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.656527 4493 generic.go:356] "Generic (PLEG): container finished" podID="0c60698d-6705-4cfc-8a08-4246536566b5" containerID="e9812bd244502fd1b362287941ea2d4286cf420d65e38d5b75bb217877484f81" exitCode=0 Nov 03 21:05:44.656718 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.656572 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt" Nov 03 21:05:44.656718 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.656602 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt" event={"ID":"0c60698d-6705-4cfc-8a08-4246536566b5","Type":"ContainerDied","Data":"e9812bd244502fd1b362287941ea2d4286cf420d65e38d5b75bb217877484f81"} Nov 03 21:05:44.656718 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.656655 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt" event={"ID":"0c60698d-6705-4cfc-8a08-4246536566b5","Type":"ContainerDied","Data":"006cf4fa724ea4eeef1f3df951b93562b182a39cc1719c4641625ca85a7e99d9"} Nov 03 21:05:44.656718 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.656676 4493 scope.go:122] "RemoveContainer" containerID="e9812bd244502fd1b362287941ea2d4286cf420d65e38d5b75bb217877484f81" Nov 03 21:05:44.658445 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.658404 4493 generic.go:356] "Generic (PLEG): container finished" podID="74a3fc99-772f-4628-b9bc-7754ae9e4b1f" containerID="a5fd2c8d623e2a31bfb5b13e8bac667dd83d8e9a709f3bc7520209d28c6bf4e7" exitCode=0 Nov 03 21:05:44.658445 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.658441 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg" event={"ID":"74a3fc99-772f-4628-b9bc-7754ae9e4b1f","Type":"ContainerDied","Data":"a5fd2c8d623e2a31bfb5b13e8bac667dd83d8e9a709f3bc7520209d28c6bf4e7"} Nov 03 21:05:44.658760 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.658481 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg" event={"ID":"74a3fc99-772f-4628-b9bc-7754ae9e4b1f","Type":"ContainerDied","Data":"dfc13d4a46120396000cbed5ee786071e2ae730e2e4a2184c7e1cfab5cf0afc5"} Nov 03 21:05:44.658760 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.658538 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg" Nov 03 21:05:44.667280 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.667265 4493 scope.go:122] "RemoveContainer" containerID="e9812bd244502fd1b362287941ea2d4286cf420d65e38d5b75bb217877484f81" Nov 03 21:05:44.667592 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:05:44.667565 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9812bd244502fd1b362287941ea2d4286cf420d65e38d5b75bb217877484f81\": container with ID starting with e9812bd244502fd1b362287941ea2d4286cf420d65e38d5b75bb217877484f81 not found: ID does not exist" containerID="e9812bd244502fd1b362287941ea2d4286cf420d65e38d5b75bb217877484f81" Nov 03 21:05:44.667698 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.667597 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9812bd244502fd1b362287941ea2d4286cf420d65e38d5b75bb217877484f81"} err="failed to get container status \"e9812bd244502fd1b362287941ea2d4286cf420d65e38d5b75bb217877484f81\": rpc error: code = NotFound desc = could not find container \"e9812bd244502fd1b362287941ea2d4286cf420d65e38d5b75bb217877484f81\": container with ID starting with e9812bd244502fd1b362287941ea2d4286cf420d65e38d5b75bb217877484f81 not found: ID does not exist" Nov 03 21:05:44.667698 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.667616 4493 scope.go:122] "RemoveContainer" containerID="a5fd2c8d623e2a31bfb5b13e8bac667dd83d8e9a709f3bc7520209d28c6bf4e7" Nov 03 21:05:44.676777 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.676760 4493 scope.go:122] "RemoveContainer" containerID="a5fd2c8d623e2a31bfb5b13e8bac667dd83d8e9a709f3bc7520209d28c6bf4e7" Nov 03 21:05:44.677035 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:05:44.677017 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5fd2c8d623e2a31bfb5b13e8bac667dd83d8e9a709f3bc7520209d28c6bf4e7\": container with ID starting with a5fd2c8d623e2a31bfb5b13e8bac667dd83d8e9a709f3bc7520209d28c6bf4e7 not found: ID does not exist" containerID="a5fd2c8d623e2a31bfb5b13e8bac667dd83d8e9a709f3bc7520209d28c6bf4e7" Nov 03 21:05:44.677082 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.677040 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5fd2c8d623e2a31bfb5b13e8bac667dd83d8e9a709f3bc7520209d28c6bf4e7"} err="failed to get container status \"a5fd2c8d623e2a31bfb5b13e8bac667dd83d8e9a709f3bc7520209d28c6bf4e7\": rpc error: code = NotFound desc = could not find container \"a5fd2c8d623e2a31bfb5b13e8bac667dd83d8e9a709f3bc7520209d28c6bf4e7\": container with ID starting with a5fd2c8d623e2a31bfb5b13e8bac667dd83d8e9a709f3bc7520209d28c6bf4e7 not found: ID does not exist" Nov 03 21:05:44.826463 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.826393 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg"] Nov 03 21:05:44.829429 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.829405 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-c63dc3-6b9784cb6c-9qqwg"] Nov 03 21:05:44.963172 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.963148 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt"] Nov 03 21:05:44.988791 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:44.988758 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c63dc3-667cc7fbff-m4zvt"] Nov 03 21:05:45.203803 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.203773 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:05:45.257772 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.257747 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:05:45.267491 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.267461 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08b80b2f-d192-42d5-9657-4d5084cdb967-kube-api-access-7vjh5\" (UniqueName: \"kubernetes.io/projected/08b80b2f-d192-42d5-9657-4d5084cdb967-kube-api-access-7vjh5\") pod \"08b80b2f-d192-42d5-9657-4d5084cdb967\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " Nov 03 21:05:45.267632 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.267545 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-credential-socket\") pod \"08b80b2f-d192-42d5-9657-4d5084cdb967\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " Nov 03 21:05:45.267723 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.267656 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-token\" (UniqueName: \"kubernetes.io/projected/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-token\") pod \"08b80b2f-d192-42d5-9657-4d5084cdb967\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " Nov 03 21:05:45.267723 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.267685 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-data\") pod \"08b80b2f-d192-42d5-9657-4d5084cdb967\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " Nov 03 21:05:45.267828 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.267759 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08b80b2f-d192-42d5-9657-4d5084cdb967-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b80b2f-d192-42d5-9657-4d5084cdb967-istiod-ca-cert\") pod \"08b80b2f-d192-42d5-9657-4d5084cdb967\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " Nov 03 21:05:45.267906 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.267892 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-workload-socket\") pod \"08b80b2f-d192-42d5-9657-4d5084cdb967\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " Nov 03 21:05:45.267965 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.267928 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-envoy\") pod \"08b80b2f-d192-42d5-9657-4d5084cdb967\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " Nov 03 21:05:45.267965 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.267939 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-credential-socket" pod "08b80b2f-d192-42d5-9657-4d5084cdb967" (UID: "08b80b2f-d192-42d5-9657-4d5084cdb967"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:45.268074 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.267978 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-podinfo\") pod \"08b80b2f-d192-42d5-9657-4d5084cdb967\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " Nov 03 21:05:45.268074 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.268027 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-workload-certs\") pod \"08b80b2f-d192-42d5-9657-4d5084cdb967\" (UID: \"08b80b2f-d192-42d5-9657-4d5084cdb967\") " Nov 03 21:05:45.268177 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.268118 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-workload-socket" pod "08b80b2f-d192-42d5-9657-4d5084cdb967" (UID: "08b80b2f-d192-42d5-9657-4d5084cdb967"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:45.268177 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.268157 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b80b2f-d192-42d5-9657-4d5084cdb967-istiod-ca-cert" pod "08b80b2f-d192-42d5-9657-4d5084cdb967" (UID: "08b80b2f-d192-42d5-9657-4d5084cdb967"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:45.268593 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.268563 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-workload-certs" pod "08b80b2f-d192-42d5-9657-4d5084cdb967" (UID: "08b80b2f-d192-42d5-9657-4d5084cdb967"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:45.268718 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.268642 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.268718 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.268651 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-data" pod "08b80b2f-d192-42d5-9657-4d5084cdb967" (UID: "08b80b2f-d192-42d5-9657-4d5084cdb967"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:45.268718 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.268665 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b80b2f-d192-42d5-9657-4d5084cdb967-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.268718 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.268681 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.270437 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.270407 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b80b2f-d192-42d5-9657-4d5084cdb967-kube-api-access-7vjh5" pod "08b80b2f-d192-42d5-9657-4d5084cdb967" (UID: "08b80b2f-d192-42d5-9657-4d5084cdb967"). InnerVolumeSpecName "kube-api-access-7vjh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:45.270530 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.270494 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-token" pod "08b80b2f-d192-42d5-9657-4d5084cdb967" (UID: "08b80b2f-d192-42d5-9657-4d5084cdb967"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:45.270736 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.270713 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-envoy" pod "08b80b2f-d192-42d5-9657-4d5084cdb967" (UID: "08b80b2f-d192-42d5-9657-4d5084cdb967"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:45.270736 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.270721 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-podinfo" pod "08b80b2f-d192-42d5-9657-4d5084cdb967" (UID: "08b80b2f-d192-42d5-9657-4d5084cdb967"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:05:45.369450 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.369415 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-workload-socket\") pod \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " Nov 03 21:05:45.369653 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.369468 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-token\" (UniqueName: \"kubernetes.io/projected/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-token\") pod \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " Nov 03 21:05:45.369653 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.369627 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-data\") pod \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " Nov 03 21:05:45.369768 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.369662 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-credential-socket\") pod \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " Nov 03 21:05:45.369768 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.369680 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-envoy\") pod \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " Nov 03 21:05:45.369768 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.369711 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-workload-certs\") pod \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " Nov 03 21:05:45.369768 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.369733 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-podinfo\") pod \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " Nov 03 21:05:45.369971 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.369759 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-workload-socket" pod "a50088fd-39b0-4118-8f90-c8e6b7fb3dd0" (UID: "a50088fd-39b0-4118-8f90-c8e6b7fb3dd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:45.369971 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.369782 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istiod-ca-cert\") pod \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " Nov 03 21:05:45.369971 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.369887 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-kube-api-access-qpn9s\" (UniqueName: \"kubernetes.io/projected/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-kube-api-access-qpn9s\") pod \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\" (UID: \"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0\") " Nov 03 21:05:45.370135 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.370058 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-data" pod "a50088fd-39b0-4118-8f90-c8e6b7fb3dd0" (UID: "a50088fd-39b0-4118-8f90-c8e6b7fb3dd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:45.370135 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.370081 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-credential-socket" pod "a50088fd-39b0-4118-8f90-c8e6b7fb3dd0" (UID: "a50088fd-39b0-4118-8f90-c8e6b7fb3dd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:45.370272 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.370248 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istiod-ca-cert" pod "a50088fd-39b0-4118-8f90-c8e6b7fb3dd0" (UID: "a50088fd-39b0-4118-8f90-c8e6b7fb3dd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:45.370500 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.370478 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-workload-certs" pod "a50088fd-39b0-4118-8f90-c8e6b7fb3dd0" (UID: "a50088fd-39b0-4118-8f90-c8e6b7fb3dd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:45.370711 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.370687 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.370804 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.370716 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.370804 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.370730 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.370804 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.370742 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.370804 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.370755 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.370804 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.370768 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.370804 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.370783 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vjh5\" (UniqueName: \"kubernetes.io/projected/08b80b2f-d192-42d5-9657-4d5084cdb967-kube-api-access-7vjh5\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.370804 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.370799 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.371034 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.370813 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.371034 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.370827 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b80b2f-d192-42d5-9657-4d5084cdb967-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.372003 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.371973 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-podinfo" pod "a50088fd-39b0-4118-8f90-c8e6b7fb3dd0" (UID: "a50088fd-39b0-4118-8f90-c8e6b7fb3dd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:05:45.372102 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.372009 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-token" pod "a50088fd-39b0-4118-8f90-c8e6b7fb3dd0" (UID: "a50088fd-39b0-4118-8f90-c8e6b7fb3dd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:45.372586 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.372563 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-envoy" pod "a50088fd-39b0-4118-8f90-c8e6b7fb3dd0" (UID: "a50088fd-39b0-4118-8f90-c8e6b7fb3dd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:45.372659 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.372587 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-kube-api-access-qpn9s" pod "a50088fd-39b0-4118-8f90-c8e6b7fb3dd0" (UID: "a50088fd-39b0-4118-8f90-c8e6b7fb3dd0"). InnerVolumeSpecName "kube-api-access-qpn9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:45.472080 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.471989 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.472303 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.472275 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.472448 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.472314 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.472448 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.472367 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpn9s\" (UniqueName: \"kubernetes.io/projected/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-kube-api-access-qpn9s\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.472448 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.472385 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.666789 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.666755 4493 generic.go:356] "Generic (PLEG): container finished" podID="08b80b2f-d192-42d5-9657-4d5084cdb967" containerID="80b183788d53186258439e88bc08ea093f0cd66f76940109eb7fe606692f3e45" exitCode=0 Nov 03 21:05:45.666930 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.666817 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" Nov 03 21:05:45.666930 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.666839 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" event={"ID":"08b80b2f-d192-42d5-9657-4d5084cdb967","Type":"ContainerDied","Data":"80b183788d53186258439e88bc08ea093f0cd66f76940109eb7fe606692f3e45"} Nov 03 21:05:45.666930 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.666902 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw" event={"ID":"08b80b2f-d192-42d5-9657-4d5084cdb967","Type":"ContainerDied","Data":"6bf2cc0a0cc91ed98b42edfcb30dda5a081788847599e8123a8fb6a91675c8eb"} Nov 03 21:05:45.666930 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.666926 4493 scope.go:122] "RemoveContainer" containerID="80b183788d53186258439e88bc08ea093f0cd66f76940109eb7fe606692f3e45" Nov 03 21:05:45.669592 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.669572 4493 generic.go:356] "Generic (PLEG): container finished" podID="a50088fd-39b0-4118-8f90-c8e6b7fb3dd0" containerID="bfad999a5615d928e6071fddaabe2b24a682bc68d423453653c33f8110c60a3e" exitCode=0 Nov 03 21:05:45.669716 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.669624 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" event={"ID":"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0","Type":"ContainerDied","Data":"bfad999a5615d928e6071fddaabe2b24a682bc68d423453653c33f8110c60a3e"} Nov 03 21:05:45.669716 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.669654 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" Nov 03 21:05:45.669716 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.669692 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz" event={"ID":"a50088fd-39b0-4118-8f90-c8e6b7fb3dd0","Type":"ContainerDied","Data":"038a5c11aca8b0c352e68c513dfe3ac47cb42c07e8908a2b000f7a238e4caa0b"} Nov 03 21:05:45.679091 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.679075 4493 scope.go:122] "RemoveContainer" containerID="80b183788d53186258439e88bc08ea093f0cd66f76940109eb7fe606692f3e45" Nov 03 21:05:45.679453 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:05:45.679431 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80b183788d53186258439e88bc08ea093f0cd66f76940109eb7fe606692f3e45\": container with ID starting with 80b183788d53186258439e88bc08ea093f0cd66f76940109eb7fe606692f3e45 not found: ID does not exist" containerID="80b183788d53186258439e88bc08ea093f0cd66f76940109eb7fe606692f3e45" Nov 03 21:05:45.679526 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.679460 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80b183788d53186258439e88bc08ea093f0cd66f76940109eb7fe606692f3e45"} err="failed to get container status \"80b183788d53186258439e88bc08ea093f0cd66f76940109eb7fe606692f3e45\": rpc error: code = NotFound desc = could not find container \"80b183788d53186258439e88bc08ea093f0cd66f76940109eb7fe606692f3e45\": container with ID starting with 80b183788d53186258439e88bc08ea093f0cd66f76940109eb7fe606692f3e45 not found: ID does not exist" Nov 03 21:05:45.679526 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.679494 4493 scope.go:122] "RemoveContainer" containerID="bfad999a5615d928e6071fddaabe2b24a682bc68d423453653c33f8110c60a3e" Nov 03 21:05:45.688915 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.688897 4493 scope.go:122] "RemoveContainer" containerID="bfad999a5615d928e6071fddaabe2b24a682bc68d423453653c33f8110c60a3e" Nov 03 21:05:45.689165 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:05:45.689146 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfad999a5615d928e6071fddaabe2b24a682bc68d423453653c33f8110c60a3e\": container with ID starting with bfad999a5615d928e6071fddaabe2b24a682bc68d423453653c33f8110c60a3e not found: ID does not exist" containerID="bfad999a5615d928e6071fddaabe2b24a682bc68d423453653c33f8110c60a3e" Nov 03 21:05:45.689244 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.689173 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfad999a5615d928e6071fddaabe2b24a682bc68d423453653c33f8110c60a3e"} err="failed to get container status \"bfad999a5615d928e6071fddaabe2b24a682bc68d423453653c33f8110c60a3e\": rpc error: code = NotFound desc = could not find container \"bfad999a5615d928e6071fddaabe2b24a682bc68d423453653c33f8110c60a3e\": container with ID starting with bfad999a5615d928e6071fddaabe2b24a682bc68d423453653c33f8110c60a3e not found: ID does not exist" Nov 03 21:05:45.837768 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.837733 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw"] Nov 03 21:05:45.889121 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.889082 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-srfsw"] Nov 03 21:05:45.955420 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.955387 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz"] Nov 03 21:05:45.964934 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:45.964907 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-5dzqz"] Nov 03 21:05:46.562475 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:46.562443 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08b80b2f-d192-42d5-9657-4d5084cdb967" path="/var/lib/kubelet/pods/08b80b2f-d192-42d5-9657-4d5084cdb967/volumes" Nov 03 21:05:46.562880 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:46.562866 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c60698d-6705-4cfc-8a08-4246536566b5" path="/var/lib/kubelet/pods/0c60698d-6705-4cfc-8a08-4246536566b5/volumes" Nov 03 21:05:46.563456 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:46.563438 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74a3fc99-772f-4628-b9bc-7754ae9e4b1f" path="/var/lib/kubelet/pods/74a3fc99-772f-4628-b9bc-7754ae9e4b1f/volumes" Nov 03 21:05:46.564306 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:46.564288 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a50088fd-39b0-4118-8f90-c8e6b7fb3dd0" path="/var/lib/kubelet/pods/a50088fd-39b0-4118-8f90-c8e6b7fb3dd0/volumes" Nov 03 21:05:51.505123 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.505082 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p"] Nov 03 21:05:51.505884 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.505860 4493 cpu_manager.go:414] "containerMap: removing container" podUID="74a3fc99-772f-4628-b9bc-7754ae9e4b1f" containerName="payload-processing" Nov 03 21:05:51.505980 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.505886 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a3fc99-772f-4628-b9bc-7754ae9e4b1f" containerName="payload-processing" Nov 03 21:05:51.505980 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.505910 4493 cpu_manager.go:414] "containerMap: removing container" podUID="a50088fd-39b0-4118-8f90-c8e6b7fb3dd0" containerName="istio-proxy" Nov 03 21:05:51.505980 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.505918 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a50088fd-39b0-4118-8f90-c8e6b7fb3dd0" containerName="istio-proxy" Nov 03 21:05:51.505980 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.505963 4493 cpu_manager.go:414] "containerMap: removing container" podUID="2c81f07e-ff42-4534-856a-01f0069a8cb2" containerName="istio-proxy" Nov 03 21:05:51.505980 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.505970 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c81f07e-ff42-4534-856a-01f0069a8cb2" containerName="istio-proxy" Nov 03 21:05:51.505980 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.505979 4493 cpu_manager.go:414] "containerMap: removing container" podUID="08b80b2f-d192-42d5-9657-4d5084cdb967" containerName="istio-proxy" Nov 03 21:05:51.506281 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.505987 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08b80b2f-d192-42d5-9657-4d5084cdb967" containerName="istio-proxy" Nov 03 21:05:51.506281 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.506012 4493 cpu_manager.go:414] "containerMap: removing container" podUID="0c60698d-6705-4cfc-8a08-4246536566b5" containerName="payload-pre-processing" Nov 03 21:05:51.506281 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.506020 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c60698d-6705-4cfc-8a08-4246536566b5" containerName="payload-pre-processing" Nov 03 21:05:51.506281 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.506197 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c60698d-6705-4cfc-8a08-4246536566b5" containerName="payload-pre-processing" Nov 03 21:05:51.506281 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.506215 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c81f07e-ff42-4534-856a-01f0069a8cb2" containerName="istio-proxy" Nov 03 21:05:51.506281 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.506227 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a3fc99-772f-4628-b9bc-7754ae9e4b1f" containerName="payload-processing" Nov 03 21:05:51.506281 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.506239 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a50088fd-39b0-4118-8f90-c8e6b7fb3dd0" containerName="istio-proxy" Nov 03 21:05:51.506281 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.506253 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="08b80b2f-d192-42d5-9657-4d5084cdb967" containerName="istio-proxy" Nov 03 21:05:51.512015 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.511992 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p" Nov 03 21:05:51.553823 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.553783 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb2be7e-d668-49e9-82ac-3475659a928a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p\" (UID: \"3cb2be7e-d668-49e9-82ac-3475659a928a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p" Nov 03 21:05:51.553823 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.553825 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95fw4\" (UniqueName: \"kubernetes.io/projected/3cb2be7e-d668-49e9-82ac-3475659a928a-kube-api-access-95fw4\") pod \"payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p\" (UID: \"3cb2be7e-d668-49e9-82ac-3475659a928a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p" Nov 03 21:05:51.562064 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.562031 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p"] Nov 03 21:05:51.618358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.610664 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7c34ff\"" Nov 03 21:05:51.618358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.610829 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7c34ff-dockercfg-zcmph\"" Nov 03 21:05:51.655791 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.655750 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb2be7e-d668-49e9-82ac-3475659a928a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p\" (UID: \"3cb2be7e-d668-49e9-82ac-3475659a928a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p" Nov 03 21:05:51.656001 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.655809 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95fw4\" (UniqueName: \"kubernetes.io/projected/3cb2be7e-d668-49e9-82ac-3475659a928a-kube-api-access-95fw4\") pod \"payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p\" (UID: \"3cb2be7e-d668-49e9-82ac-3475659a928a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p" Nov 03 21:05:51.660356 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.657288 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb2be7e-d668-49e9-82ac-3475659a928a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p\" (UID: \"3cb2be7e-d668-49e9-82ac-3475659a928a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p" Nov 03 21:05:51.660356 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.659149 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s"] Nov 03 21:05:51.660356 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.659826 4493 cpu_manager.go:414] "containerMap: removing container" podUID="2c81f07e-ff42-4534-856a-01f0069a8cb2" containerName="istio-proxy" Nov 03 21:05:51.660356 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.659841 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c81f07e-ff42-4534-856a-01f0069a8cb2" containerName="istio-proxy" Nov 03 21:05:51.670419 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.668718 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c81f07e-ff42-4534-856a-01f0069a8cb2" containerName="istio-proxy" Nov 03 21:05:51.716357 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.695857 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s" Nov 03 21:05:51.716357 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.711483 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s"] Nov 03 21:05:51.730819 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.730788 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95fw4\" (UniqueName: \"kubernetes.io/projected/3cb2be7e-d668-49e9-82ac-3475659a928a-kube-api-access-95fw4\") pod \"payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p\" (UID: \"3cb2be7e-d668-49e9-82ac-3475659a928a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p" Nov 03 21:05:51.755370 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:05:51.755272 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:05:51.757195 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.757164 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26hc4\" (UniqueName: \"kubernetes.io/projected/1a6f7bb0-d357-43cb-9ad0-165987580c0b-kube-api-access-26hc4\") pod \"payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s\" (UID: \"1a6f7bb0-d357-43cb-9ad0-165987580c0b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s" Nov 03 21:05:51.757478 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.757457 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6f7bb0-d357-43cb-9ad0-165987580c0b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s\" (UID: \"1a6f7bb0-d357-43cb-9ad0-165987580c0b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s" Nov 03 21:05:51.825962 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.825933 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p" Nov 03 21:05:51.859380 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.859312 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6f7bb0-d357-43cb-9ad0-165987580c0b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s\" (UID: \"1a6f7bb0-d357-43cb-9ad0-165987580c0b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s" Nov 03 21:05:51.859529 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.859509 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26hc4\" (UniqueName: \"kubernetes.io/projected/1a6f7bb0-d357-43cb-9ad0-165987580c0b-kube-api-access-26hc4\") pod \"payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s\" (UID: \"1a6f7bb0-d357-43cb-9ad0-165987580c0b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s" Nov 03 21:05:51.860034 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:51.860011 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6f7bb0-d357-43cb-9ad0-165987580c0b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s\" (UID: \"1a6f7bb0-d357-43cb-9ad0-165987580c0b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s" Nov 03 21:05:52.025500 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:52.025373 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26hc4\" (UniqueName: \"kubernetes.io/projected/1a6f7bb0-d357-43cb-9ad0-165987580c0b-kube-api-access-26hc4\") pod \"payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s\" (UID: \"1a6f7bb0-d357-43cb-9ad0-165987580c0b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s" Nov 03 21:05:52.033959 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:52.033690 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s" Nov 03 21:05:52.209364 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:52.209317 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p"] Nov 03 21:05:52.211169 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:05:52.211142 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb2be7e_d668_49e9_82ac_3475659a928a.slice/crio-2c06eb354606d9da8f9ddc0bb8de52367ab9276a0def5874cec624d234f40941 WatchSource:0}: Error finding container 2c06eb354606d9da8f9ddc0bb8de52367ab9276a0def5874cec624d234f40941: Status 404 returned error can't find the container with id 2c06eb354606d9da8f9ddc0bb8de52367ab9276a0def5874cec624d234f40941 Nov 03 21:05:52.426634 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:52.426603 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s"] Nov 03 21:05:52.426766 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:05:52.426607 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6f7bb0_d357_43cb_9ad0_165987580c0b.slice/crio-f41560c93352fc4ef40f608106cbc0600b21d0047681ec213f539e63a5f54216 WatchSource:0}: Error finding container f41560c93352fc4ef40f608106cbc0600b21d0047681ec213f539e63a5f54216: Status 404 returned error can't find the container with id f41560c93352fc4ef40f608106cbc0600b21d0047681ec213f539e63a5f54216 Nov 03 21:05:52.761735 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:52.761696 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p" event={"ID":"3cb2be7e-d668-49e9-82ac-3475659a928a","Type":"ContainerStarted","Data":"2b892d7c7369a314c6cd8056c7e8649ee4b772a78ddd547047a9add5fd86e43a"} Nov 03 21:05:52.762316 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:52.761760 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p" event={"ID":"3cb2be7e-d668-49e9-82ac-3475659a928a","Type":"ContainerStarted","Data":"2c06eb354606d9da8f9ddc0bb8de52367ab9276a0def5874cec624d234f40941"} Nov 03 21:05:52.762757 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:52.762734 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p" Nov 03 21:05:52.764258 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:52.764233 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s" event={"ID":"1a6f7bb0-d357-43cb-9ad0-165987580c0b","Type":"ContainerStarted","Data":"54152a273046453430849c4b0dfc5f3ac0c5a83d51d508e4464df5067d9872ed"} Nov 03 21:05:52.764424 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:52.764280 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s" event={"ID":"1a6f7bb0-d357-43cb-9ad0-165987580c0b","Type":"ContainerStarted","Data":"f41560c93352fc4ef40f608106cbc0600b21d0047681ec213f539e63a5f54216"} Nov 03 21:05:52.764863 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:52.764843 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s" Nov 03 21:05:52.882840 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:52.882778 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p" podStartSLOduration=1.882763977 podStartE2EDuration="1.882763977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:52.878845329 +0000 UTC m=+2224.436835839" watchObservedRunningTime="2026-11-03 21:05:52.882763977 +0000 UTC m=+2224.440754476" Nov 03 21:05:53.004104 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:05:53.004034 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s" podStartSLOduration=2.004014728 podStartE2EDuration="2.004014728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:53.000752323 +0000 UTC m=+2224.558742849" watchObservedRunningTime="2026-11-03 21:05:53.004014728 +0000 UTC m=+2224.562005228" Nov 03 21:06:02.264946 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:02.264909 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c"] Nov 03 21:06:02.269386 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:02.269368 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c" Nov 03 21:06:02.377357 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:02.349649 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c"] Nov 03 21:06:02.377357 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:02.371958 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb648add-5f46-4562-83f2-eb0703c98d55-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c\" (UID: \"cb648add-5f46-4562-83f2-eb0703c98d55\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c" Nov 03 21:06:02.377357 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:02.372074 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb6sn\" (UniqueName: \"kubernetes.io/projected/cb648add-5f46-4562-83f2-eb0703c98d55-kube-api-access-wb6sn\") pod \"payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c\" (UID: \"cb648add-5f46-4562-83f2-eb0703c98d55\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c" Nov 03 21:06:02.377357 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:02.372430 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-211d68bf\"" Nov 03 21:06:02.377357 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:02.372600 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-211d68bf-dockercfg-l9rj8\"" Nov 03 21:06:02.414590 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:02.414551 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm"] Nov 03 21:06:02.419787 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:02.419758 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm" Nov 03 21:06:02.472462 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:02.472430 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm"] Nov 03 21:06:02.473424 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:02.473308 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd-plugins-config-volume\") pod \"payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm\" (UID: \"ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd\") " pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm" Nov 03 21:06:02.473565 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:02.473544 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb648add-5f46-4562-83f2-eb0703c98d55-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c\" (UID: \"cb648add-5f46-4562-83f2-eb0703c98d55\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c" Nov 03 21:06:02.473655 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:02.473637 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb6sn\" (UniqueName: \"kubernetes.io/projected/cb648add-5f46-4562-83f2-eb0703c98d55-kube-api-access-wb6sn\") pod \"payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c\" (UID: \"cb648add-5f46-4562-83f2-eb0703c98d55\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c" Nov 03 21:06:02.473780 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:02.473703 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwnb5\" (UniqueName: \"kubernetes.io/projected/ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd-kube-api-access-jwnb5\") pod \"payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm\" (UID: \"ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd\") " pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm" Nov 03 21:06:02.474585 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:02.474563 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb648add-5f46-4562-83f2-eb0703c98d55-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c\" (UID: \"cb648add-5f46-4562-83f2-eb0703c98d55\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c" Nov 03 21:06:02.533305 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:02.533229 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb6sn\" (UniqueName: \"kubernetes.io/projected/cb648add-5f46-4562-83f2-eb0703c98d55-kube-api-access-wb6sn\") pod \"payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c\" (UID: \"cb648add-5f46-4562-83f2-eb0703c98d55\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c" Nov 03 21:06:02.574601 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:02.574568 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd-plugins-config-volume\") pod \"payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm\" (UID: \"ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd\") " pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm" Nov 03 21:06:02.574831 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:02.574709 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwnb5\" (UniqueName: \"kubernetes.io/projected/ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd-kube-api-access-jwnb5\") pod \"payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm\" (UID: \"ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd\") " pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm" Nov 03 21:06:02.575263 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:02.575239 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd-plugins-config-volume\") pod \"payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm\" (UID: \"ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd\") " pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm" Nov 03 21:06:02.583152 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:02.583124 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c" Nov 03 21:06:02.633810 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:02.633774 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwnb5\" (UniqueName: \"kubernetes.io/projected/ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd-kube-api-access-jwnb5\") pod \"payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm\" (UID: \"ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd\") " pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm" Nov 03 21:06:02.752212 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:02.752171 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm" Nov 03 21:06:02.979359 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:02.979298 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c"] Nov 03 21:06:03.136591 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:06:03.136564 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2b8063_56a5_4adf_a1eb_a5b7e1b301cd.slice/crio-41c2a4c551c2d8799cff4828514da298fbda82948b10ca8d872025162d34eddb WatchSource:0}: Error finding container 41c2a4c551c2d8799cff4828514da298fbda82948b10ca8d872025162d34eddb: Status 404 returned error can't find the container with id 41c2a4c551c2d8799cff4828514da298fbda82948b10ca8d872025162d34eddb Nov 03 21:06:03.137145 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:03.137123 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm"] Nov 03 21:06:03.843507 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:03.843468 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c" event={"ID":"cb648add-5f46-4562-83f2-eb0703c98d55","Type":"ContainerStarted","Data":"410cdca36a940890def1a6fa110b240364d82357c14f5543b04047b61ad914e0"} Nov 03 21:06:03.843988 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:03.843535 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c" event={"ID":"cb648add-5f46-4562-83f2-eb0703c98d55","Type":"ContainerStarted","Data":"3796b046ed7df7fda668b0c659c87b6f1e5ce66e941a5fa165752bfbfd290a15"} Nov 03 21:06:03.843988 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:03.843677 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c" Nov 03 21:06:03.845274 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:03.845251 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm" event={"ID":"ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd","Type":"ContainerStarted","Data":"716870a9dda8f6d264e262241fac05041ff005cc120aff0237f4a4de00e48028"} Nov 03 21:06:03.845415 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:03.845300 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm" Nov 03 21:06:03.845415 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:03.845316 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm" event={"ID":"ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd","Type":"ContainerStarted","Data":"41c2a4c551c2d8799cff4828514da298fbda82948b10ca8d872025162d34eddb"} Nov 03 21:06:03.964101 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:03.964049 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c" podStartSLOduration=1.964036384 podStartE2EDuration="1.964036384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:03.962813775 +0000 UTC m=+2235.520804276" watchObservedRunningTime="2026-11-03 21:06:03.964036384 +0000 UTC m=+2235.522026882" Nov 03 21:06:04.084889 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:04.084823 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm" podStartSLOduration=2.084797672 podStartE2EDuration="2.084797672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:04.079076251 +0000 UTC m=+2235.637066770" watchObservedRunningTime="2026-11-03 21:06:04.084797672 +0000 UTC m=+2235.642788178" Nov 03 21:06:04.785208 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:04.785166 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p" Nov 03 21:06:04.785445 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:04.785251 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s" Nov 03 21:06:12.650151 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.650119 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws"] Nov 03 21:06:12.656532 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.656500 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.820304 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.820270 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c17f9c-7151-410b-99e4-e09c03c16171-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.820510 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.820369 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00c17f9c-7151-410b-99e4-e09c03c16171-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.820510 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.820436 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.820588 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.820557 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69dpz\" (UniqueName: \"kubernetes.io/projected/00c17f9c-7151-410b-99e4-e09c03c16171-kube-api-access-69dpz\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.820720 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.820706 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.820762 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.820748 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.820802 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.820789 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00c17f9c-7151-410b-99e4-e09c03c16171-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.820837 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.820822 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.820877 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.820865 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.856984 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.856950 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws"] Nov 03 21:06:12.922223 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.922140 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.922223 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.922196 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c17f9c-7151-410b-99e4-e09c03c16171-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.922223 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.922222 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00c17f9c-7151-410b-99e4-e09c03c16171-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.922559 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.922244 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.922559 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.922304 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69dpz\" (UniqueName: \"kubernetes.io/projected/00c17f9c-7151-410b-99e4-e09c03c16171-kube-api-access-69dpz\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.922559 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.922483 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.922559 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.922531 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.922775 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.922559 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00c17f9c-7151-410b-99e4-e09c03c16171-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.922775 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.922587 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.922775 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.922604 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.922959 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.922881 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.922959 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.922928 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.923072 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.922959 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.923124 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.923067 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00c17f9c-7151-410b-99e4-e09c03c16171-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.924804 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.924777 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.925392 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.925371 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00c17f9c-7151-410b-99e4-e09c03c16171-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.980838 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.980793 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c17f9c-7151-410b-99e4-e09c03c16171-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:12.981031 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:12.980941 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69dpz\" (UniqueName: \"kubernetes.io/projected/00c17f9c-7151-410b-99e4-e09c03c16171-kube-api-access-69dpz\") pod \"maas-default-gateway-openshift-default-696f47c6c-mppws\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:13.275272 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:13.275191 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:13.719731 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:06:13.719703 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c17f9c_7151_410b_99e4_e09c03c16171.slice/crio-89afaae80551a2144e281472a885d68e5ee7d2a474d2f0f6f819b03232394485 WatchSource:0}: Error finding container 89afaae80551a2144e281472a885d68e5ee7d2a474d2f0f6f819b03232394485: Status 404 returned error can't find the container with id 89afaae80551a2144e281472a885d68e5ee7d2a474d2f0f6f819b03232394485 Nov 03 21:06:13.720196 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:13.720158 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws"] Nov 03 21:06:13.722406 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:13.722368 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:06:13.722516 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:13.722442 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:06:13.722516 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:13.722472 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:06:13.914848 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:13.914809 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" event={"ID":"00c17f9c-7151-410b-99e4-e09c03c16171","Type":"ContainerStarted","Data":"c487b03955d5bb80c93c91cda3525c4e8041b9eabaa94fab1ed70332646043ae"} Nov 03 21:06:13.915014 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:13.914872 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" event={"ID":"00c17f9c-7151-410b-99e4-e09c03c16171","Type":"ContainerStarted","Data":"89afaae80551a2144e281472a885d68e5ee7d2a474d2f0f6f819b03232394485"} Nov 03 21:06:14.037518 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:14.037395 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" podStartSLOduration=2.037377136 podStartE2EDuration="2.037377136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:14.034792106 +0000 UTC m=+2245.592782607" watchObservedRunningTime="2026-11-03 21:06:14.037377136 +0000 UTC m=+2245.595367636" Nov 03 21:06:14.278590 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:14.278539 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:14.280479 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:14.280444 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-mppws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 21:06:14.280670 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:14.280505 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" podUID="00c17f9c-7151-410b-99e4-e09c03c16171" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 21:06:14.854514 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:14.854483 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm" Nov 03 21:06:14.855011 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:14.854564 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c" Nov 03 21:06:15.275984 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:15.275899 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-mppws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 21:06:15.275984 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:15.275960 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" podUID="00c17f9c-7151-410b-99e4-e09c03c16171" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 21:06:16.070762 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:16.070719 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr"] Nov 03 21:06:16.071420 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:16.071387 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" podUID="c3be0d37-42be-4b25-846b-d8538f590441" containerName="istio-proxy" containerID="cri-o://998d8632d44a6e74b97cd33b89a524d24f21aac51ad7e6835436a00043135e2a" gracePeriod=30 Nov 03 21:06:16.275951 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:16.275921 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-mppws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 21:06:16.276127 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:16.275977 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" podUID="00c17f9c-7151-410b-99e4-e09c03c16171" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 21:06:17.275746 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:17.275708 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-mppws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 21:06:17.276230 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:17.275781 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" podUID="00c17f9c-7151-410b-99e4-e09c03c16171" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 21:06:18.276200 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:18.276164 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-mppws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 21:06:18.276719 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:18.276233 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" podUID="00c17f9c-7151-410b-99e4-e09c03c16171" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 21:06:19.276345 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:19.276305 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-mppws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 21:06:19.276742 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:19.276389 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" podUID="00c17f9c-7151-410b-99e4-e09c03c16171" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 21:06:20.276412 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:20.276377 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-mppws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 21:06:20.276871 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:20.276437 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" podUID="00c17f9c-7151-410b-99e4-e09c03c16171" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 21:06:21.290649 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.287179 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-mppws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 21:06:21.290649 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.287240 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" podUID="00c17f9c-7151-410b-99e4-e09c03c16171" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 21:06:21.415710 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.415682 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 21:06:21.494821 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.494785 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3be0d37-42be-4b25-846b-d8538f590441-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3be0d37-42be-4b25-846b-d8538f590441-istio-podinfo\") pod \"c3be0d37-42be-4b25-846b-d8538f590441\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " Nov 03 21:06:21.495039 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.494892 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-workload-certs\") pod \"c3be0d37-42be-4b25-846b-d8538f590441\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " Nov 03 21:06:21.495039 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.495014 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-istio-data\") pod \"c3be0d37-42be-4b25-846b-d8538f590441\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " Nov 03 21:06:21.495162 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.495087 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3be0d37-42be-4b25-846b-d8538f590441-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3be0d37-42be-4b25-846b-d8538f590441-istiod-ca-cert\") pod \"c3be0d37-42be-4b25-846b-d8538f590441\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " Nov 03 21:06:21.495162 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.495127 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-credential-socket\") pod \"c3be0d37-42be-4b25-846b-d8538f590441\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " Nov 03 21:06:21.495279 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.495189 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3be0d37-42be-4b25-846b-d8538f590441-istio-token\" (UniqueName: \"kubernetes.io/projected/c3be0d37-42be-4b25-846b-d8538f590441-istio-token\") pod \"c3be0d37-42be-4b25-846b-d8538f590441\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " Nov 03 21:06:21.495279 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.495215 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-workload-certs" pod "c3be0d37-42be-4b25-846b-d8538f590441" (UID: "c3be0d37-42be-4b25-846b-d8538f590441"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:21.495279 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.495264 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-istio-envoy\") pod \"c3be0d37-42be-4b25-846b-d8538f590441\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " Nov 03 21:06:21.495463 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.495282 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-istio-data" pod "c3be0d37-42be-4b25-846b-d8538f590441" (UID: "c3be0d37-42be-4b25-846b-d8538f590441"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:21.495463 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.495351 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-credential-socket" pod "c3be0d37-42be-4b25-846b-d8538f590441" (UID: "c3be0d37-42be-4b25-846b-d8538f590441"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:21.495463 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.495367 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-workload-socket\") pod \"c3be0d37-42be-4b25-846b-d8538f590441\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " Nov 03 21:06:21.495463 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.495417 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3be0d37-42be-4b25-846b-d8538f590441-kube-api-access-zl56q\" (UniqueName: \"kubernetes.io/projected/c3be0d37-42be-4b25-846b-d8538f590441-kube-api-access-zl56q\") pod \"c3be0d37-42be-4b25-846b-d8538f590441\" (UID: \"c3be0d37-42be-4b25-846b-d8538f590441\") " Nov 03 21:06:21.495688 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.495617 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3be0d37-42be-4b25-846b-d8538f590441-istiod-ca-cert" pod "c3be0d37-42be-4b25-846b-d8538f590441" (UID: "c3be0d37-42be-4b25-846b-d8538f590441"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:06:21.495889 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.495863 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-workload-socket" pod "c3be0d37-42be-4b25-846b-d8538f590441" (UID: "c3be0d37-42be-4b25-846b-d8538f590441"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:21.496446 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.496424 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:21.496548 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.496451 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:21.496548 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.496467 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:21.496548 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.496481 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3be0d37-42be-4b25-846b-d8538f590441-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:21.496548 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.496496 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:21.497896 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.497870 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-istio-envoy" pod "c3be0d37-42be-4b25-846b-d8538f590441" (UID: "c3be0d37-42be-4b25-846b-d8538f590441"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:21.498076 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.497922 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3be0d37-42be-4b25-846b-d8538f590441-istio-podinfo" pod "c3be0d37-42be-4b25-846b-d8538f590441" (UID: "c3be0d37-42be-4b25-846b-d8538f590441"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:06:21.498294 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.498277 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3be0d37-42be-4b25-846b-d8538f590441-istio-token" pod "c3be0d37-42be-4b25-846b-d8538f590441" (UID: "c3be0d37-42be-4b25-846b-d8538f590441"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:21.498618 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.498600 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3be0d37-42be-4b25-846b-d8538f590441-kube-api-access-zl56q" pod "c3be0d37-42be-4b25-846b-d8538f590441" (UID: "c3be0d37-42be-4b25-846b-d8538f590441"). InnerVolumeSpecName "kube-api-access-zl56q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:21.597455 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.597424 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3be0d37-42be-4b25-846b-d8538f590441-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:21.597455 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.597451 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3be0d37-42be-4b25-846b-d8538f590441-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:21.597455 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.597461 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl56q\" (UniqueName: \"kubernetes.io/projected/c3be0d37-42be-4b25-846b-d8538f590441-kube-api-access-zl56q\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:21.597772 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.597470 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3be0d37-42be-4b25-846b-d8538f590441-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:21.757670 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:06:21.757540 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:06:21.928712 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.928674 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8"] Nov 03 21:06:21.982649 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.982620 4493 generic.go:356] "Generic (PLEG): container finished" podID="c3be0d37-42be-4b25-846b-d8538f590441" containerID="998d8632d44a6e74b97cd33b89a524d24f21aac51ad7e6835436a00043135e2a" exitCode=0 Nov 03 21:06:21.982824 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.982671 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" Nov 03 21:06:21.982824 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.982699 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" event={"ID":"c3be0d37-42be-4b25-846b-d8538f590441","Type":"ContainerDied","Data":"998d8632d44a6e74b97cd33b89a524d24f21aac51ad7e6835436a00043135e2a"} Nov 03 21:06:21.982824 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.982757 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr" event={"ID":"c3be0d37-42be-4b25-846b-d8538f590441","Type":"ContainerDied","Data":"55d263ee570a343e71e56d4c53600170a86a81541d5015ad0ae4c438f51c119c"} Nov 03 21:06:21.982824 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.982787 4493 scope.go:122] "RemoveContainer" containerID="998d8632d44a6e74b97cd33b89a524d24f21aac51ad7e6835436a00043135e2a" Nov 03 21:06:21.997021 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:21.996997 4493 scope.go:122] "RemoveContainer" containerID="cc23d43d708717c675b978b9771987a15b127af157120d844b8cbfb7b8120eea" Nov 03 21:06:22.009781 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:22.009624 4493 scope.go:122] "RemoveContainer" containerID="998d8632d44a6e74b97cd33b89a524d24f21aac51ad7e6835436a00043135e2a" Nov 03 21:06:22.010067 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:06:22.010035 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"998d8632d44a6e74b97cd33b89a524d24f21aac51ad7e6835436a00043135e2a\": container with ID starting with 998d8632d44a6e74b97cd33b89a524d24f21aac51ad7e6835436a00043135e2a not found: ID does not exist" containerID="998d8632d44a6e74b97cd33b89a524d24f21aac51ad7e6835436a00043135e2a" Nov 03 21:06:22.010187 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:22.010084 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"998d8632d44a6e74b97cd33b89a524d24f21aac51ad7e6835436a00043135e2a"} err="failed to get container status \"998d8632d44a6e74b97cd33b89a524d24f21aac51ad7e6835436a00043135e2a\": rpc error: code = NotFound desc = could not find container \"998d8632d44a6e74b97cd33b89a524d24f21aac51ad7e6835436a00043135e2a\": container with ID starting with 998d8632d44a6e74b97cd33b89a524d24f21aac51ad7e6835436a00043135e2a not found: ID does not exist" Nov 03 21:06:22.010187 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:22.010123 4493 scope.go:122] "RemoveContainer" containerID="cc23d43d708717c675b978b9771987a15b127af157120d844b8cbfb7b8120eea" Nov 03 21:06:22.014724 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:06:22.014597 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc23d43d708717c675b978b9771987a15b127af157120d844b8cbfb7b8120eea\": container with ID starting with cc23d43d708717c675b978b9771987a15b127af157120d844b8cbfb7b8120eea not found: ID does not exist" containerID="cc23d43d708717c675b978b9771987a15b127af157120d844b8cbfb7b8120eea" Nov 03 21:06:22.014900 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:22.014868 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc23d43d708717c675b978b9771987a15b127af157120d844b8cbfb7b8120eea"} err="failed to get container status \"cc23d43d708717c675b978b9771987a15b127af157120d844b8cbfb7b8120eea\": rpc error: code = NotFound desc = could not find container \"cc23d43d708717c675b978b9771987a15b127af157120d844b8cbfb7b8120eea\": container with ID starting with cc23d43d708717c675b978b9771987a15b127af157120d844b8cbfb7b8120eea not found: ID does not exist" Nov 03 21:06:22.041144 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:22.041106 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv"] Nov 03 21:06:22.164794 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:22.164762 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr"] Nov 03 21:06:22.170290 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:22.170256 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-h6mnr"] Nov 03 21:06:22.390082 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:22.390052 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:22.391464 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:22.391432 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:22.392311 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:22.392288 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:06:22.564690 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:22.564658 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3be0d37-42be-4b25-846b-d8538f590441" path="/var/lib/kubelet/pods/c3be0d37-42be-4b25-846b-d8538f590441/volumes" Nov 03 21:06:26.929390 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:26.929316 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8" podUID="d4d5ceb0-75f0-420a-8e62-26eadb0ee960" containerName="payload-processing" containerID="cri-o://fc734675b5bba411266e830bb569aa1f484cfcea0209281e80dfc551344ce3a7" gracePeriod=25 Nov 03 21:06:27.041690 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:27.041644 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv" podUID="43ec7143-f92e-406d-82e4-9c71bb5ff7ac" containerName="payload-pre-processing" containerID="cri-o://bcd76ddcddb59773709fc7a56a799bd2c74339f191b8f987142727f1ead01f2c" gracePeriod=25 Nov 03 21:06:27.201127 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:27.201067 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8" Nov 03 21:06:27.277083 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:27.277046 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4d5ceb0-75f0-420a-8e62-26eadb0ee960-kube-api-access-pxtk2\" (UniqueName: \"kubernetes.io/projected/d4d5ceb0-75f0-420a-8e62-26eadb0ee960-kube-api-access-pxtk2\") pod \"d4d5ceb0-75f0-420a-8e62-26eadb0ee960\" (UID: \"d4d5ceb0-75f0-420a-8e62-26eadb0ee960\") " Nov 03 21:06:27.277260 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:27.277149 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4d5ceb0-75f0-420a-8e62-26eadb0ee960-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4d5ceb0-75f0-420a-8e62-26eadb0ee960-plugins-config-volume\") pod \"d4d5ceb0-75f0-420a-8e62-26eadb0ee960\" (UID: \"d4d5ceb0-75f0-420a-8e62-26eadb0ee960\") " Nov 03 21:06:27.277514 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:27.277488 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d5ceb0-75f0-420a-8e62-26eadb0ee960-plugins-config-volume" pod "d4d5ceb0-75f0-420a-8e62-26eadb0ee960" (UID: "d4d5ceb0-75f0-420a-8e62-26eadb0ee960"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:06:27.277669 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:27.277651 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4d5ceb0-75f0-420a-8e62-26eadb0ee960-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:27.307179 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:27.307128 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d5ceb0-75f0-420a-8e62-26eadb0ee960-kube-api-access-pxtk2" pod "d4d5ceb0-75f0-420a-8e62-26eadb0ee960" (UID: "d4d5ceb0-75f0-420a-8e62-26eadb0ee960"). InnerVolumeSpecName "kube-api-access-pxtk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:27.326372 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:27.326346 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv" Nov 03 21:06:27.379033 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:27.379001 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxtk2\" (UniqueName: \"kubernetes.io/projected/d4d5ceb0-75f0-420a-8e62-26eadb0ee960-kube-api-access-pxtk2\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:27.479938 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:27.479863 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ec7143-f92e-406d-82e4-9c71bb5ff7ac-kube-api-access-9qjzs\" (UniqueName: \"kubernetes.io/projected/43ec7143-f92e-406d-82e4-9c71bb5ff7ac-kube-api-access-9qjzs\") pod \"43ec7143-f92e-406d-82e4-9c71bb5ff7ac\" (UID: \"43ec7143-f92e-406d-82e4-9c71bb5ff7ac\") " Nov 03 21:06:27.480078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:27.480016 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ec7143-f92e-406d-82e4-9c71bb5ff7ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ec7143-f92e-406d-82e4-9c71bb5ff7ac-plugins-config-volume\") pod \"43ec7143-f92e-406d-82e4-9c71bb5ff7ac\" (UID: \"43ec7143-f92e-406d-82e4-9c71bb5ff7ac\") " Nov 03 21:06:27.480409 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:27.480383 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ec7143-f92e-406d-82e4-9c71bb5ff7ac-plugins-config-volume" pod "43ec7143-f92e-406d-82e4-9c71bb5ff7ac" (UID: "43ec7143-f92e-406d-82e4-9c71bb5ff7ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:06:27.480549 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:27.480531 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ec7143-f92e-406d-82e4-9c71bb5ff7ac-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:27.482138 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:27.482114 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ec7143-f92e-406d-82e4-9c71bb5ff7ac-kube-api-access-9qjzs" pod "43ec7143-f92e-406d-82e4-9c71bb5ff7ac" (UID: "43ec7143-f92e-406d-82e4-9c71bb5ff7ac"). InnerVolumeSpecName "kube-api-access-9qjzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:27.581949 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:27.581919 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qjzs\" (UniqueName: \"kubernetes.io/projected/43ec7143-f92e-406d-82e4-9c71bb5ff7ac-kube-api-access-9qjzs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:28.030256 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:28.030227 4493 generic.go:356] "Generic (PLEG): container finished" podID="43ec7143-f92e-406d-82e4-9c71bb5ff7ac" containerID="bcd76ddcddb59773709fc7a56a799bd2c74339f191b8f987142727f1ead01f2c" exitCode=0 Nov 03 21:06:28.030728 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:28.030271 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv" Nov 03 21:06:28.030728 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:28.030310 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv" event={"ID":"43ec7143-f92e-406d-82e4-9c71bb5ff7ac","Type":"ContainerDied","Data":"bcd76ddcddb59773709fc7a56a799bd2c74339f191b8f987142727f1ead01f2c"} Nov 03 21:06:28.030728 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:28.030377 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv" event={"ID":"43ec7143-f92e-406d-82e4-9c71bb5ff7ac","Type":"ContainerDied","Data":"4da11d8554a9f857c98b2916f9dd4b7e45cab824145407b33904201785af589b"} Nov 03 21:06:28.030728 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:28.030401 4493 scope.go:122] "RemoveContainer" containerID="bcd76ddcddb59773709fc7a56a799bd2c74339f191b8f987142727f1ead01f2c" Nov 03 21:06:28.032000 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:28.031977 4493 generic.go:356] "Generic (PLEG): container finished" podID="d4d5ceb0-75f0-420a-8e62-26eadb0ee960" containerID="fc734675b5bba411266e830bb569aa1f484cfcea0209281e80dfc551344ce3a7" exitCode=0 Nov 03 21:06:28.032123 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:28.032081 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8" event={"ID":"d4d5ceb0-75f0-420a-8e62-26eadb0ee960","Type":"ContainerDied","Data":"fc734675b5bba411266e830bb569aa1f484cfcea0209281e80dfc551344ce3a7"} Nov 03 21:06:28.032123 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:28.032097 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8" Nov 03 21:06:28.032123 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:28.032113 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8" event={"ID":"d4d5ceb0-75f0-420a-8e62-26eadb0ee960","Type":"ContainerDied","Data":"f47080c01530f52de930645cab3210104dd76b2901ca7fa7b1feb6a51ba9ad36"} Nov 03 21:06:28.041955 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:28.041929 4493 scope.go:122] "RemoveContainer" containerID="bcd76ddcddb59773709fc7a56a799bd2c74339f191b8f987142727f1ead01f2c" Nov 03 21:06:28.042238 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:06:28.042217 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcd76ddcddb59773709fc7a56a799bd2c74339f191b8f987142727f1ead01f2c\": container with ID starting with bcd76ddcddb59773709fc7a56a799bd2c74339f191b8f987142727f1ead01f2c not found: ID does not exist" containerID="bcd76ddcddb59773709fc7a56a799bd2c74339f191b8f987142727f1ead01f2c" Nov 03 21:06:28.042326 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:28.042251 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcd76ddcddb59773709fc7a56a799bd2c74339f191b8f987142727f1ead01f2c"} err="failed to get container status \"bcd76ddcddb59773709fc7a56a799bd2c74339f191b8f987142727f1ead01f2c\": rpc error: code = NotFound desc = could not find container \"bcd76ddcddb59773709fc7a56a799bd2c74339f191b8f987142727f1ead01f2c\": container with ID starting with bcd76ddcddb59773709fc7a56a799bd2c74339f191b8f987142727f1ead01f2c not found: ID does not exist" Nov 03 21:06:28.042326 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:28.042268 4493 scope.go:122] "RemoveContainer" containerID="fc734675b5bba411266e830bb569aa1f484cfcea0209281e80dfc551344ce3a7" Nov 03 21:06:28.050917 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:28.050896 4493 scope.go:122] "RemoveContainer" containerID="fc734675b5bba411266e830bb569aa1f484cfcea0209281e80dfc551344ce3a7" Nov 03 21:06:28.051170 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:06:28.051152 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc734675b5bba411266e830bb569aa1f484cfcea0209281e80dfc551344ce3a7\": container with ID starting with fc734675b5bba411266e830bb569aa1f484cfcea0209281e80dfc551344ce3a7 not found: ID does not exist" containerID="fc734675b5bba411266e830bb569aa1f484cfcea0209281e80dfc551344ce3a7" Nov 03 21:06:28.051232 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:28.051177 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc734675b5bba411266e830bb569aa1f484cfcea0209281e80dfc551344ce3a7"} err="failed to get container status \"fc734675b5bba411266e830bb569aa1f484cfcea0209281e80dfc551344ce3a7\": rpc error: code = NotFound desc = could not find container \"fc734675b5bba411266e830bb569aa1f484cfcea0209281e80dfc551344ce3a7\": container with ID starting with fc734675b5bba411266e830bb569aa1f484cfcea0209281e80dfc551344ce3a7 not found: ID does not exist" Nov 03 21:06:28.199137 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:28.199064 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv"] Nov 03 21:06:28.251522 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:28.251487 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-88bc35-66c8f68675-6j5nv"] Nov 03 21:06:28.362474 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:28.362440 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8"] Nov 03 21:06:28.365469 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:28.365448 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-88bc35-969d4ffcc-5t9h8"] Nov 03 21:06:28.562640 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:28.562602 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43ec7143-f92e-406d-82e4-9c71bb5ff7ac" path="/var/lib/kubelet/pods/43ec7143-f92e-406d-82e4-9c71bb5ff7ac/volumes" Nov 03 21:06:28.563138 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:28.563116 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4d5ceb0-75f0-420a-8e62-26eadb0ee960" path="/var/lib/kubelet/pods/d4d5ceb0-75f0-420a-8e62-26eadb0ee960/volumes" Nov 03 21:06:48.256988 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:48.256939 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd"] Nov 03 21:06:48.257505 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:48.257231 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" podUID="46f74fa9-7638-44fa-b303-813d37808176" containerName="istio-proxy" containerID="cri-o://86658c8dbf134c8a6d69b430266834e285da6350c2c29582a1cae6a4a6554f10" gracePeriod=30 Nov 03 21:06:51.303151 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.303118 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5"] Nov 03 21:06:51.303712 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.303697 4493 cpu_manager.go:414] "containerMap: removing container" podUID="d4d5ceb0-75f0-420a-8e62-26eadb0ee960" containerName="payload-processing" Nov 03 21:06:51.303756 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.303713 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4d5ceb0-75f0-420a-8e62-26eadb0ee960" containerName="payload-processing" Nov 03 21:06:51.303756 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.303728 4493 cpu_manager.go:414] "containerMap: removing container" podUID="43ec7143-f92e-406d-82e4-9c71bb5ff7ac" containerName="payload-pre-processing" Nov 03 21:06:51.303756 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.303733 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ec7143-f92e-406d-82e4-9c71bb5ff7ac" containerName="payload-pre-processing" Nov 03 21:06:51.303756 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.303755 4493 cpu_manager.go:414] "containerMap: removing container" podUID="c3be0d37-42be-4b25-846b-d8538f590441" containerName="istio-proxy" Nov 03 21:06:51.303890 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.303759 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3be0d37-42be-4b25-846b-d8538f590441" containerName="istio-proxy" Nov 03 21:06:51.303890 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.303770 4493 cpu_manager.go:414] "containerMap: removing container" podUID="c3be0d37-42be-4b25-846b-d8538f590441" containerName="istio-proxy" Nov 03 21:06:51.303890 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.303774 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3be0d37-42be-4b25-846b-d8538f590441" containerName="istio-proxy" Nov 03 21:06:51.303890 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.303851 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3be0d37-42be-4b25-846b-d8538f590441" containerName="istio-proxy" Nov 03 21:06:51.303890 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.303862 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4d5ceb0-75f0-420a-8e62-26eadb0ee960" containerName="payload-processing" Nov 03 21:06:51.303890 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.303872 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ec7143-f92e-406d-82e4-9c71bb5ff7ac" containerName="payload-pre-processing" Nov 03 21:06:51.308862 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.308846 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5" Nov 03 21:06:51.360919 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.360893 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5"] Nov 03 21:06:51.408621 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.408586 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c14420-serving-cert\"" Nov 03 21:06:51.467160 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.467130 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-maas-api-tls\") pod \"maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5\" (UID: \"9f86bc65-604a-44f2-bbe5-8fdc21fa38a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5" Nov 03 21:06:51.467378 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.467258 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv7ht\" (UniqueName: \"kubernetes.io/projected/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-kube-api-access-kv7ht\") pod \"maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5\" (UID: \"9f86bc65-604a-44f2-bbe5-8fdc21fa38a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5" Nov 03 21:06:51.467378 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.467315 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-metrics-certs\") pod \"maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5\" (UID: \"9f86bc65-604a-44f2-bbe5-8fdc21fa38a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5" Nov 03 21:06:51.467570 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.467553 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5\" (UID: \"9f86bc65-604a-44f2-bbe5-8fdc21fa38a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5" Nov 03 21:06:51.568865 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.568780 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5\" (UID: \"9f86bc65-604a-44f2-bbe5-8fdc21fa38a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5" Nov 03 21:06:51.569029 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.568868 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-maas-api-tls\") pod \"maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5\" (UID: \"9f86bc65-604a-44f2-bbe5-8fdc21fa38a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5" Nov 03 21:06:51.569029 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.568931 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv7ht\" (UniqueName: \"kubernetes.io/projected/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-kube-api-access-kv7ht\") pod \"maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5\" (UID: \"9f86bc65-604a-44f2-bbe5-8fdc21fa38a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5" Nov 03 21:06:51.569029 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.568966 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-metrics-certs\") pod \"maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5\" (UID: \"9f86bc65-604a-44f2-bbe5-8fdc21fa38a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5" Nov 03 21:06:51.569554 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.569530 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5\" (UID: \"9f86bc65-604a-44f2-bbe5-8fdc21fa38a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5" Nov 03 21:06:51.571985 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.571961 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-maas-api-tls\") pod \"maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5\" (UID: \"9f86bc65-604a-44f2-bbe5-8fdc21fa38a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5" Nov 03 21:06:51.572128 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.572101 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-metrics-certs\") pod \"maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5\" (UID: \"9f86bc65-604a-44f2-bbe5-8fdc21fa38a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5" Nov 03 21:06:51.631186 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.631154 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv7ht\" (UniqueName: \"kubernetes.io/projected/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-kube-api-access-kv7ht\") pod \"maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5\" (UID: \"9f86bc65-604a-44f2-bbe5-8fdc21fa38a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5" Nov 03 21:06:51.756497 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:06:51.756458 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:06:51.922625 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:51.922546 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5" Nov 03 21:06:52.302788 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:06:52.302755 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f86bc65_604a_44f2_bbe5_8fdc21fa38a9.slice/crio-d8eb35e5d618489deaeb38a435c71bb1b46b0c8d470af8d081bd7515e3cf5a1a WatchSource:0}: Error finding container d8eb35e5d618489deaeb38a435c71bb1b46b0c8d470af8d081bd7515e3cf5a1a: Status 404 returned error can't find the container with id d8eb35e5d618489deaeb38a435c71bb1b46b0c8d470af8d081bd7515e3cf5a1a Nov 03 21:06:52.302788 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:52.302776 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5"] Nov 03 21:06:52.304222 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:52.304205 4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:06:53.252378 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.252313 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5" event={"ID":"9f86bc65-604a-44f2-bbe5-8fdc21fa38a9","Type":"ContainerStarted","Data":"d8eb35e5d618489deaeb38a435c71bb1b46b0c8d470af8d081bd7515e3cf5a1a"} Nov 03 21:06:53.546292 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.546259 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:06:53.698208 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.698172 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-istio-data\") pod \"46f74fa9-7638-44fa-b303-813d37808176\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " Nov 03 21:06:53.698430 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.698314 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46f74fa9-7638-44fa-b303-813d37808176-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f74fa9-7638-44fa-b303-813d37808176-istio-podinfo\") pod \"46f74fa9-7638-44fa-b303-813d37808176\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " Nov 03 21:06:53.698430 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.698418 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46f74fa9-7638-44fa-b303-813d37808176-kube-api-access-qf7jx\" (UniqueName: \"kubernetes.io/projected/46f74fa9-7638-44fa-b303-813d37808176-kube-api-access-qf7jx\") pod \"46f74fa9-7638-44fa-b303-813d37808176\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " Nov 03 21:06:53.698571 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.698433 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-istio-data" pod "46f74fa9-7638-44fa-b303-813d37808176" (UID: "46f74fa9-7638-44fa-b303-813d37808176"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:53.698571 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.698502 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46f74fa9-7638-44fa-b303-813d37808176-istio-token\" (UniqueName: \"kubernetes.io/projected/46f74fa9-7638-44fa-b303-813d37808176-istio-token\") pod \"46f74fa9-7638-44fa-b303-813d37808176\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " Nov 03 21:06:53.698571 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.698555 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-workload-socket\") pod \"46f74fa9-7638-44fa-b303-813d37808176\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " Nov 03 21:06:53.698728 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.698587 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-istio-envoy\") pod \"46f74fa9-7638-44fa-b303-813d37808176\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " Nov 03 21:06:53.698728 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.698655 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-credential-socket\") pod \"46f74fa9-7638-44fa-b303-813d37808176\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " Nov 03 21:06:53.698828 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.698751 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46f74fa9-7638-44fa-b303-813d37808176-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f74fa9-7638-44fa-b303-813d37808176-istiod-ca-cert\") pod \"46f74fa9-7638-44fa-b303-813d37808176\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " Nov 03 21:06:53.698828 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.698800 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-workload-certs\") pod \"46f74fa9-7638-44fa-b303-813d37808176\" (UID: \"46f74fa9-7638-44fa-b303-813d37808176\") " Nov 03 21:06:53.698828 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.698799 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-workload-socket" pod "46f74fa9-7638-44fa-b303-813d37808176" (UID: "46f74fa9-7638-44fa-b303-813d37808176"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:53.698976 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.698954 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-credential-socket" pod "46f74fa9-7638-44fa-b303-813d37808176" (UID: "46f74fa9-7638-44fa-b303-813d37808176"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:53.699167 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.699141 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46f74fa9-7638-44fa-b303-813d37808176-istiod-ca-cert" pod "46f74fa9-7638-44fa-b303-813d37808176" (UID: "46f74fa9-7638-44fa-b303-813d37808176"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:06:53.699516 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.699252 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-workload-certs" pod "46f74fa9-7638-44fa-b303-813d37808176" (UID: "46f74fa9-7638-44fa-b303-813d37808176"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:53.699609 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.699586 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f74fa9-7638-44fa-b303-813d37808176-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:53.699688 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.699609 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:53.699688 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.699625 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:53.699688 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.699648 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:53.699688 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.699663 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:53.700819 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.700791 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f74fa9-7638-44fa-b303-813d37808176-istio-token" pod "46f74fa9-7638-44fa-b303-813d37808176" (UID: "46f74fa9-7638-44fa-b303-813d37808176"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:53.700819 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.700802 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46f74fa9-7638-44fa-b303-813d37808176-istio-podinfo" pod "46f74fa9-7638-44fa-b303-813d37808176" (UID: "46f74fa9-7638-44fa-b303-813d37808176"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:06:53.700819 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.700809 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-istio-envoy" pod "46f74fa9-7638-44fa-b303-813d37808176" (UID: "46f74fa9-7638-44fa-b303-813d37808176"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:53.701071 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.701056 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f74fa9-7638-44fa-b303-813d37808176-kube-api-access-qf7jx" pod "46f74fa9-7638-44fa-b303-813d37808176" (UID: "46f74fa9-7638-44fa-b303-813d37808176"). InnerVolumeSpecName "kube-api-access-qf7jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:53.801262 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.801227 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f74fa9-7638-44fa-b303-813d37808176-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:53.801262 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.801254 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f74fa9-7638-44fa-b303-813d37808176-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:53.801262 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.801264 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f74fa9-7638-44fa-b303-813d37808176-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:53.801607 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:53.801276 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf7jx\" (UniqueName: \"kubernetes.io/projected/46f74fa9-7638-44fa-b303-813d37808176-kube-api-access-qf7jx\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:54.293023 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:54.292764 4493 generic.go:356] "Generic (PLEG): container finished" podID="46f74fa9-7638-44fa-b303-813d37808176" containerID="86658c8dbf134c8a6d69b430266834e285da6350c2c29582a1cae6a4a6554f10" exitCode=0 Nov 03 21:06:54.293187 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:54.293020 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" event={"ID":"46f74fa9-7638-44fa-b303-813d37808176","Type":"ContainerDied","Data":"86658c8dbf134c8a6d69b430266834e285da6350c2c29582a1cae6a4a6554f10"} Nov 03 21:06:54.293187 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:54.293077 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" event={"ID":"46f74fa9-7638-44fa-b303-813d37808176","Type":"ContainerDied","Data":"f91cb11a92c521d09787163112b6f0756610281b687731ac4d2bf62eb45f4b57"} Nov 03 21:06:54.293187 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:54.293108 4493 scope.go:122] "RemoveContainer" containerID="86658c8dbf134c8a6d69b430266834e285da6350c2c29582a1cae6a4a6554f10" Nov 03 21:06:54.293389 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:54.293266 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd" Nov 03 21:06:54.296949 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:54.296897 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5" event={"ID":"9f86bc65-604a-44f2-bbe5-8fdc21fa38a9","Type":"ContainerStarted","Data":"6afd8a22beb88a183351b7c1e357d9f63f93e27cba0cb313462bd0adf1f78ac6"} Nov 03 21:06:54.301514 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:54.300037 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5" Nov 03 21:06:54.312620 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:54.312599 4493 scope.go:122] "RemoveContainer" containerID="389ff94293cd7d3a448630714c20f13a9699dd5cf4ac30bc030c18b5aad79b94" Nov 03 21:06:54.323965 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:54.323939 4493 scope.go:122] "RemoveContainer" containerID="86658c8dbf134c8a6d69b430266834e285da6350c2c29582a1cae6a4a6554f10" Nov 03 21:06:54.324311 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:06:54.324286 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86658c8dbf134c8a6d69b430266834e285da6350c2c29582a1cae6a4a6554f10\": container with ID starting with 86658c8dbf134c8a6d69b430266834e285da6350c2c29582a1cae6a4a6554f10 not found: ID does not exist" containerID="86658c8dbf134c8a6d69b430266834e285da6350c2c29582a1cae6a4a6554f10" Nov 03 21:06:54.324447 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:54.324321 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86658c8dbf134c8a6d69b430266834e285da6350c2c29582a1cae6a4a6554f10"} err="failed to get container status \"86658c8dbf134c8a6d69b430266834e285da6350c2c29582a1cae6a4a6554f10\": rpc error: code = NotFound desc = could not find container \"86658c8dbf134c8a6d69b430266834e285da6350c2c29582a1cae6a4a6554f10\": container with ID starting with 86658c8dbf134c8a6d69b430266834e285da6350c2c29582a1cae6a4a6554f10 not found: ID does not exist" Nov 03 21:06:54.324447 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:54.324381 4493 scope.go:122] "RemoveContainer" containerID="389ff94293cd7d3a448630714c20f13a9699dd5cf4ac30bc030c18b5aad79b94" Nov 03 21:06:54.324698 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:06:54.324678 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"389ff94293cd7d3a448630714c20f13a9699dd5cf4ac30bc030c18b5aad79b94\": container with ID starting with 389ff94293cd7d3a448630714c20f13a9699dd5cf4ac30bc030c18b5aad79b94 not found: ID does not exist" containerID="389ff94293cd7d3a448630714c20f13a9699dd5cf4ac30bc030c18b5aad79b94" Nov 03 21:06:54.324747 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:54.324703 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"389ff94293cd7d3a448630714c20f13a9699dd5cf4ac30bc030c18b5aad79b94"} err="failed to get container status \"389ff94293cd7d3a448630714c20f13a9699dd5cf4ac30bc030c18b5aad79b94\": rpc error: code = NotFound desc = could not find container \"389ff94293cd7d3a448630714c20f13a9699dd5cf4ac30bc030c18b5aad79b94\": container with ID starting with 389ff94293cd7d3a448630714c20f13a9699dd5cf4ac30bc030c18b5aad79b94 not found: ID does not exist" Nov 03 21:06:54.467648 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:54.467579 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5" podStartSLOduration=2.504998643 podStartE2EDuration="3.467561556s" totalImagesPullingTime="962.562913ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:54.465938699 +0000 UTC m=+2286.023929200" watchObservedRunningTime="2026-11-03 21:06:54.467561556 +0000 UTC m=+2286.025552058" Nov 03 21:06:54.585311 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:54.585225 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd"] Nov 03 21:06:54.588356 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:54.588313 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-c63dc3-openshift-default-74cf4d998f-tx7cd"] Nov 03 21:06:56.070647 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:56.070605 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j"] Nov 03 21:06:56.071406 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:56.071384 4493 cpu_manager.go:414] "containerMap: removing container" podUID="46f74fa9-7638-44fa-b303-813d37808176" containerName="istio-proxy" Nov 03 21:06:56.071406 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:56.071405 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46f74fa9-7638-44fa-b303-813d37808176" containerName="istio-proxy" Nov 03 21:06:56.071594 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:56.071426 4493 cpu_manager.go:414] "containerMap: removing container" podUID="46f74fa9-7638-44fa-b303-813d37808176" containerName="istio-proxy" Nov 03 21:06:56.071594 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:56.071433 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46f74fa9-7638-44fa-b303-813d37808176" containerName="istio-proxy" Nov 03 21:06:56.071714 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:56.071650 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="46f74fa9-7638-44fa-b303-813d37808176" containerName="istio-proxy" Nov 03 21:06:56.071714 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:56.071666 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="46f74fa9-7638-44fa-b303-813d37808176" containerName="istio-proxy" Nov 03 21:06:56.071714 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:56.071687 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3be0d37-42be-4b25-846b-d8538f590441" containerName="istio-proxy" Nov 03 21:06:56.076789 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:56.076767 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j" Nov 03 21:06:56.130754 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:56.130724 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j"] Nov 03 21:06:56.176162 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:56.176137 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-211d68bf-serving-cert\"" Nov 03 21:06:56.233871 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:56.233699 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zncw\" (UniqueName: \"kubernetes.io/projected/bc04227c-da6e-421c-b258-135ebe1e519e-kube-api-access-8zncw\") pod \"maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j\" (UID: \"bc04227c-da6e-421c-b258-135ebe1e519e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j" Nov 03 21:06:56.233871 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:56.233782 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc04227c-da6e-421c-b258-135ebe1e519e-maas-api-tls\") pod \"maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j\" (UID: \"bc04227c-da6e-421c-b258-135ebe1e519e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j" Nov 03 21:06:56.234124 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:56.233904 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc04227c-da6e-421c-b258-135ebe1e519e-openshift-service-ca\") pod \"maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j\" (UID: \"bc04227c-da6e-421c-b258-135ebe1e519e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j" Nov 03 21:06:56.234124 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:56.234075 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc04227c-da6e-421c-b258-135ebe1e519e-metrics-certs\") pod \"maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j\" (UID: \"bc04227c-da6e-421c-b258-135ebe1e519e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j" Nov 03 21:06:56.336960 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:56.336099 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zncw\" (UniqueName: \"kubernetes.io/projected/bc04227c-da6e-421c-b258-135ebe1e519e-kube-api-access-8zncw\") pod \"maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j\" (UID: \"bc04227c-da6e-421c-b258-135ebe1e519e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j" Nov 03 21:06:56.336960 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:56.336197 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc04227c-da6e-421c-b258-135ebe1e519e-maas-api-tls\") pod \"maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j\" (UID: \"bc04227c-da6e-421c-b258-135ebe1e519e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j" Nov 03 21:06:56.336960 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:56.336304 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc04227c-da6e-421c-b258-135ebe1e519e-openshift-service-ca\") pod \"maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j\" (UID: \"bc04227c-da6e-421c-b258-135ebe1e519e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j" Nov 03 21:06:56.336960 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:56.336623 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc04227c-da6e-421c-b258-135ebe1e519e-metrics-certs\") pod \"maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j\" (UID: \"bc04227c-da6e-421c-b258-135ebe1e519e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j" Nov 03 21:06:56.337379 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:06:56.337357 4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-serving-cert: secret "maas-api-e2e-mt-211d68bf-serving-cert" not found Nov 03 21:06:56.337591 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:06:56.337572 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc04227c-da6e-421c-b258-135ebe1e519e-maas-api-tls podName:bc04227c-da6e-421c-b258-135ebe1e519e nodeName:}" failed. No retries permitted until 2026-11-03 21:06:56.837521922 +0000 UTC m=+2288.395512400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bc04227c-da6e-421c-b258-135ebe1e519e-maas-api-tls") pod "maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j" (UID: "bc04227c-da6e-421c-b258-135ebe1e519e") : secret "maas-api-e2e-mt-211d68bf-serving-cert" not found Nov 03 21:06:56.338847 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:56.338804 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc04227c-da6e-421c-b258-135ebe1e519e-openshift-service-ca\") pod \"maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j\" (UID: \"bc04227c-da6e-421c-b258-135ebe1e519e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j" Nov 03 21:06:56.340647 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:56.340619 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc04227c-da6e-421c-b258-135ebe1e519e-metrics-certs\") pod \"maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j\" (UID: \"bc04227c-da6e-421c-b258-135ebe1e519e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j" Nov 03 21:06:56.397991 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:56.397958 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zncw\" (UniqueName: \"kubernetes.io/projected/bc04227c-da6e-421c-b258-135ebe1e519e-kube-api-access-8zncw\") pod \"maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j\" (UID: \"bc04227c-da6e-421c-b258-135ebe1e519e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j" Nov 03 21:06:56.562938 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:56.562901 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46f74fa9-7638-44fa-b303-813d37808176" path="/var/lib/kubelet/pods/46f74fa9-7638-44fa-b303-813d37808176/volumes" Nov 03 21:06:56.843853 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:56.843824 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc04227c-da6e-421c-b258-135ebe1e519e-maas-api-tls\") pod \"maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j\" (UID: \"bc04227c-da6e-421c-b258-135ebe1e519e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j" Nov 03 21:06:56.846325 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:56.846305 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc04227c-da6e-421c-b258-135ebe1e519e-maas-api-tls\") pod \"maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j\" (UID: \"bc04227c-da6e-421c-b258-135ebe1e519e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j" Nov 03 21:06:56.990705 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:56.990672 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j" Nov 03 21:06:57.372027 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:57.372005 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j"] Nov 03 21:06:57.372365 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:06:57.372240 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc04227c_da6e_421c_b258_135ebe1e519e.slice/crio-b8d182eb8f0842b2002ee561620ba1c40aaead09cc0a36524c00bdddcda3d14a WatchSource:0}: Error finding container b8d182eb8f0842b2002ee561620ba1c40aaead09cc0a36524c00bdddcda3d14a: Status 404 returned error can't find the container with id b8d182eb8f0842b2002ee561620ba1c40aaead09cc0a36524c00bdddcda3d14a Nov 03 21:06:58.333944 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:58.333910 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j" event={"ID":"bc04227c-da6e-421c-b258-135ebe1e519e","Type":"ContainerStarted","Data":"b8d182eb8f0842b2002ee561620ba1c40aaead09cc0a36524c00bdddcda3d14a"} Nov 03 21:06:59.342801 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:59.342766 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j" event={"ID":"bc04227c-da6e-421c-b258-135ebe1e519e","Type":"ContainerStarted","Data":"5ed251b06cc3759bea07888a96482fb1abc5fc9f0f78995dcb3ee66a4fd5e710"} Nov 03 21:06:59.343274 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:59.342858 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j" Nov 03 21:06:59.459999 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:06:59.459948 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j" podStartSLOduration=2.552021234 podStartE2EDuration="3.459934394s" totalImagesPullingTime="907.91316ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:59.459538193 +0000 UTC m=+2291.017528692" watchObservedRunningTime="2026-11-03 21:06:59.459934394 +0000 UTC m=+2291.017924892" Nov 03 21:07:00.341990 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:00.341935 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5" Nov 03 21:07:05.364688 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:05.364653 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j" Nov 03 21:07:05.614520 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:05.614488 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj"] Nov 03 21:07:05.618916 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:05.618872 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj" Nov 03 21:07:05.672247 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:05.672213 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj"] Nov 03 21:07:05.674891 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:05.674864 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7c34ff-serving-cert\"" Nov 03 21:07:05.762473 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:05.762442 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a47f518-c3db-4150-99ee-00061ae2ae1e-metrics-certs\") pod \"maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj\" (UID: \"5a47f518-c3db-4150-99ee-00061ae2ae1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj" Nov 03 21:07:05.762473 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:05.762471 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm4x8\" (UniqueName: \"kubernetes.io/projected/5a47f518-c3db-4150-99ee-00061ae2ae1e-kube-api-access-pm4x8\") pod \"maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj\" (UID: \"5a47f518-c3db-4150-99ee-00061ae2ae1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj" Nov 03 21:07:05.762882 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:05.762731 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a47f518-c3db-4150-99ee-00061ae2ae1e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj\" (UID: \"5a47f518-c3db-4150-99ee-00061ae2ae1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj" Nov 03 21:07:05.762882 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:05.762819 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a47f518-c3db-4150-99ee-00061ae2ae1e-maas-api-tls\") pod \"maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj\" (UID: \"5a47f518-c3db-4150-99ee-00061ae2ae1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj" Nov 03 21:07:05.864749 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:05.864712 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a47f518-c3db-4150-99ee-00061ae2ae1e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj\" (UID: \"5a47f518-c3db-4150-99ee-00061ae2ae1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj" Nov 03 21:07:05.865837 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:05.864792 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a47f518-c3db-4150-99ee-00061ae2ae1e-maas-api-tls\") pod \"maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj\" (UID: \"5a47f518-c3db-4150-99ee-00061ae2ae1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj" Nov 03 21:07:05.865837 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:05.864884 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a47f518-c3db-4150-99ee-00061ae2ae1e-metrics-certs\") pod \"maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj\" (UID: \"5a47f518-c3db-4150-99ee-00061ae2ae1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj" Nov 03 21:07:05.865837 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:05.864906 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm4x8\" (UniqueName: \"kubernetes.io/projected/5a47f518-c3db-4150-99ee-00061ae2ae1e-kube-api-access-pm4x8\") pod \"maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj\" (UID: \"5a47f518-c3db-4150-99ee-00061ae2ae1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj" Nov 03 21:07:05.865837 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:07:05.865534 4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-serving-cert: secret "maas-api-e2e-inf-a-7c34ff-serving-cert" not found Nov 03 21:07:05.865837 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:05.865620 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a47f518-c3db-4150-99ee-00061ae2ae1e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj\" (UID: \"5a47f518-c3db-4150-99ee-00061ae2ae1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj" Nov 03 21:07:05.865837 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:07:05.865636 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a47f518-c3db-4150-99ee-00061ae2ae1e-maas-api-tls podName:5a47f518-c3db-4150-99ee-00061ae2ae1e nodeName:}" failed. No retries permitted until 2026-11-03 21:07:06.365615655 +0000 UTC m=+2297.923606151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5a47f518-c3db-4150-99ee-00061ae2ae1e-maas-api-tls") pod "maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj" (UID: "5a47f518-c3db-4150-99ee-00061ae2ae1e") : secret "maas-api-e2e-inf-a-7c34ff-serving-cert" not found Nov 03 21:07:05.868633 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:05.868607 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a47f518-c3db-4150-99ee-00061ae2ae1e-metrics-certs\") pod \"maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj\" (UID: \"5a47f518-c3db-4150-99ee-00061ae2ae1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj" Nov 03 21:07:05.924070 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:05.924001 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm4x8\" (UniqueName: \"kubernetes.io/projected/5a47f518-c3db-4150-99ee-00061ae2ae1e-kube-api-access-pm4x8\") pod \"maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj\" (UID: \"5a47f518-c3db-4150-99ee-00061ae2ae1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj" Nov 03 21:07:06.377856 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:06.377819 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a47f518-c3db-4150-99ee-00061ae2ae1e-maas-api-tls\") pod \"maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj\" (UID: \"5a47f518-c3db-4150-99ee-00061ae2ae1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj" Nov 03 21:07:06.380451 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:06.380426 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a47f518-c3db-4150-99ee-00061ae2ae1e-maas-api-tls\") pod \"maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj\" (UID: \"5a47f518-c3db-4150-99ee-00061ae2ae1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj" Nov 03 21:07:06.532597 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:06.532559 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj" Nov 03 21:07:06.913055 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:06.913021 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj"] Nov 03 21:07:06.913202 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:07:06.913056 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a47f518_c3db_4150_99ee_00061ae2ae1e.slice/crio-b05b7bc63baa7df345fe1bfe6e1e6dcc3c72b3bab7edee804802c533cb27e874 WatchSource:0}: Error finding container b05b7bc63baa7df345fe1bfe6e1e6dcc3c72b3bab7edee804802c533cb27e874: Status 404 returned error can't find the container with id b05b7bc63baa7df345fe1bfe6e1e6dcc3c72b3bab7edee804802c533cb27e874 Nov 03 21:07:07.404471 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:07.404437 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj" event={"ID":"5a47f518-c3db-4150-99ee-00061ae2ae1e","Type":"ContainerStarted","Data":"b05b7bc63baa7df345fe1bfe6e1e6dcc3c72b3bab7edee804802c533cb27e874"} Nov 03 21:07:08.412838 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:08.412803 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj" event={"ID":"5a47f518-c3db-4150-99ee-00061ae2ae1e","Type":"ContainerStarted","Data":"21acb298c6d2242d7b6779a25dcfe2e2e3a5ee3c929ad238692ba96abb7a43a7"} Nov 03 21:07:08.413224 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:08.412868 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj" Nov 03 21:07:08.532003 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:08.531952 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj" podStartSLOduration=2.638009741 podStartE2EDuration="3.531937419s" totalImagesPullingTime="893.927678ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:08.530241585 +0000 UTC m=+2300.088232106" watchObservedRunningTime="2026-11-03 21:07:08.531937419 +0000 UTC m=+2300.089927927" Nov 03 21:07:14.425211 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:14.425185 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj" Nov 03 21:07:21.745251 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:07:21.745214 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:07:51.748573 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:07:51.748536 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:07:53.282801 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:53.281447 4493 patch_prober.go:29] interesting pod/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": read tcp 10.133.0.2:33778->10.133.0.127:15021: read: connection reset by peer" start-of-body= Nov 03 21:07:53.282801 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:53.281533 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" podUID="b5b5e9e0-94df-40e7-b031-15c11babc655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": read tcp 10.133.0.2:33778->10.133.0.127:15021: read: connection reset by peer" Nov 03 21:07:53.759151 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:53.759116 4493 generic.go:356] "Generic (PLEG): container finished" podID="b5b5e9e0-94df-40e7-b031-15c11babc655" containerID="ff8f9c1a71bb1c053be98b46bc6bb774a160468595af4f4fed0660ea97293553" exitCode=0 Nov 03 21:07:53.759359 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:53.759193 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" event={"ID":"b5b5e9e0-94df-40e7-b031-15c11babc655","Type":"ContainerDied","Data":"ff8f9c1a71bb1c053be98b46bc6bb774a160468595af4f4fed0660ea97293553"} Nov 03 21:07:53.759771 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:53.759755 4493 scope.go:122] "RemoveContainer" containerID="ff8f9c1a71bb1c053be98b46bc6bb774a160468595af4f4fed0660ea97293553" Nov 03 21:07:53.761885 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:53.761849 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:07:53.761984 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:53.761946 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:07:53.762025 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:53.761996 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:07:54.768187 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:54.768151 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" event={"ID":"b5b5e9e0-94df-40e7-b031-15c11babc655","Type":"ContainerStarted","Data":"34f3ccb6b6d54ea25a128595fb12b7d9d0fb465e14cd9b64fe59041ef4bbf1d9"} Nov 03 21:07:54.808534 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:54.808496 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:07:54.808810 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:54.808786 4493 patch_prober.go:29] interesting pod/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 21:07:54.808915 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:54.808846 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" podUID="b5b5e9e0-94df-40e7-b031-15c11babc655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 21:07:55.803367 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:55.802667 4493 patch_prober.go:29] interesting pod/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 21:07:55.803367 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:55.802732 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" podUID="b5b5e9e0-94df-40e7-b031-15c11babc655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 21:07:56.802704 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:56.802668 4493 patch_prober.go:29] interesting pod/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 21:07:56.803040 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:56.803005 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" podUID="b5b5e9e0-94df-40e7-b031-15c11babc655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 21:07:57.808806 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:57.808772 4493 patch_prober.go:29] interesting pod/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 21:07:57.809466 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:57.809431 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" podUID="b5b5e9e0-94df-40e7-b031-15c11babc655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 21:07:58.805747 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:58.805282 4493 patch_prober.go:29] interesting pod/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 21:07:58.805747 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:58.805375 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" podUID="b5b5e9e0-94df-40e7-b031-15c11babc655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 21:07:59.800372 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:59.800327 4493 patch_prober.go:29] interesting pod/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 21:07:59.800801 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:07:59.800393 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" podUID="b5b5e9e0-94df-40e7-b031-15c11babc655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 21:08:00.804198 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:00.804164 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:08:00.804799 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:00.804562 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:08:00.805300 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:00.805284 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:08:02.632356 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.627749 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f"] Nov 03 21:08:02.650537 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.650507 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.707552 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.707519 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.707552 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.707556 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.707835 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.707653 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.707835 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.707771 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.707835 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.707824 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs4lv\" (UniqueName: \"kubernetes.io/projected/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-kube-api-access-xs4lv\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.708001 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.707881 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.708001 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.707965 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.708120 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.708007 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.708120 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.708032 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.755471 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.755442 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f"] Nov 03 21:08:02.809431 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.809400 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.809621 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.809453 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.809621 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.809521 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.809621 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.809552 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.809800 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.809627 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.809800 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.809738 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.809800 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.809787 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs4lv\" (UniqueName: \"kubernetes.io/projected/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-kube-api-access-xs4lv\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.809962 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.809824 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.809962 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.809928 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.809962 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.809955 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.810217 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.810195 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.810435 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.810413 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.810681 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.810658 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.811041 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.811020 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.813408 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.813386 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.813786 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.813768 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.867926 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.867897 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.868677 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.868657 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs4lv\" (UniqueName: \"kubernetes.io/projected/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-kube-api-access-xs4lv\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-65r9f\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:02.965371 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:02.965286 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:03.420831 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:08:03.420802 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c1d4d0_289d_4028_a3c5_b16bbeb87b40.slice/crio-c32fb4556c3463c4fad8fb4c3e45a3dffba834a29e8eb2748540eaddaacb0cca WatchSource:0}: Error finding container c32fb4556c3463c4fad8fb4c3e45a3dffba834a29e8eb2748540eaddaacb0cca: Status 404 returned error can't find the container with id c32fb4556c3463c4fad8fb4c3e45a3dffba834a29e8eb2748540eaddaacb0cca Nov 03 21:08:03.421068 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:03.421048 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f"] Nov 03 21:08:03.423299 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:03.423272 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:08:03.423426 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:03.423351 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:08:03.423426 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:03.423402 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:08:03.849462 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:03.849426 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" event={"ID":"49c1d4d0-289d-4028-a3c5-b16bbeb87b40","Type":"ContainerStarted","Data":"5edae911cd698dc7e0f4ada4ec0f2012a59409673a613caf815a12fbd110df38"} Nov 03 21:08:03.849842 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:03.849496 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" event={"ID":"49c1d4d0-289d-4028-a3c5-b16bbeb87b40","Type":"ContainerStarted","Data":"c32fb4556c3463c4fad8fb4c3e45a3dffba834a29e8eb2748540eaddaacb0cca"} Nov 03 21:08:03.965803 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:03.965770 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:04.010110 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:04.010055 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" podStartSLOduration=2.010041727 podStartE2EDuration="2.010041727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:04.00904402 +0000 UTC m=+2355.567034538" watchObservedRunningTime="2026-11-03 21:08:04.010041727 +0000 UTC m=+2355.568032226" Nov 03 21:08:04.967777 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:04.967746 4493 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-65r9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 03 21:08:04.968153 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:04.967795 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" podUID="49c1d4d0-289d-4028-a3c5-b16bbeb87b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 03 21:08:05.965618 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:05.965580 4493 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-65r9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 03 21:08:05.965792 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:05.965644 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" podUID="49c1d4d0-289d-4028-a3c5-b16bbeb87b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 03 21:08:06.965968 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:06.965934 4493 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-65r9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 03 21:08:06.966485 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:06.965988 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" podUID="49c1d4d0-289d-4028-a3c5-b16bbeb87b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 03 21:08:07.965763 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:07.965723 4493 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-65r9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 03 21:08:07.965945 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:07.965797 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" podUID="49c1d4d0-289d-4028-a3c5-b16bbeb87b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 03 21:08:08.966157 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:08.966120 4493 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-65r9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 03 21:08:08.966585 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:08.966188 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" podUID="49c1d4d0-289d-4028-a3c5-b16bbeb87b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 03 21:08:09.966400 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:09.966363 4493 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-65r9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 03 21:08:09.966879 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:09.966423 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" podUID="49c1d4d0-289d-4028-a3c5-b16bbeb87b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 03 21:08:10.971821 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:10.971791 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:10.972304 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:10.972281 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:10.973119 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:10.973102 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:08:17.738804 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.738771 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv"] Nov 03 21:08:17.743105 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.743086 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:17.885202 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.885168 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:17.885409 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.885212 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:17.885409 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.885251 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:17.885409 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.885382 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:17.885588 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.885425 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:17.885588 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.885516 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:17.885588 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.885543 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:17.885716 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.885609 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:17.885716 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.885651 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-847jx\" (UniqueName: \"kubernetes.io/projected/7ec5039b-2ca6-4099-9919-12f4b42af9c7-kube-api-access-847jx\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:17.946873 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.946837 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv"] Nov 03 21:08:17.987004 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.986973 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:17.987183 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.987011 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:17.987183 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.987035 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:17.987183 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.987171 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-847jx\" (UniqueName: \"kubernetes.io/projected/7ec5039b-2ca6-4099-9919-12f4b42af9c7-kube-api-access-847jx\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:17.987489 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.987356 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:17.987489 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.987400 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:17.987489 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.987442 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:17.987489 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.987457 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:17.987489 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.987486 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:17.987785 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.987571 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:17.987785 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.987607 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:17.987785 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.987768 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:17.988015 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.987844 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:17.988015 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.987989 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:17.990185 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.990129 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:17.990317 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:17.990296 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:18.047373 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:18.047320 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:18.047544 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:18.047514 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-847jx\" (UniqueName: \"kubernetes.io/projected/7ec5039b-2ca6-4099-9919-12f4b42af9c7-kube-api-access-847jx\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m2tfv\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:18.069837 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:18.069813 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:18.512615 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:18.512585 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv"] Nov 03 21:08:18.515080 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:08:18.515044 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec5039b_2ca6_4099_9919_12f4b42af9c7.slice/crio-78c9f1cef8b2dc9360f038aafd810923d4874af421b7e9b607d47ee67bb30aca WatchSource:0}: Error finding container 78c9f1cef8b2dc9360f038aafd810923d4874af421b7e9b607d47ee67bb30aca: Status 404 returned error can't find the container with id 78c9f1cef8b2dc9360f038aafd810923d4874af421b7e9b607d47ee67bb30aca Nov 03 21:08:18.519452 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:18.518564 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:08:18.519452 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:18.518661 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:08:18.519452 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:18.518704 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:08:19.003616 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:19.003571 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" event={"ID":"7ec5039b-2ca6-4099-9919-12f4b42af9c7","Type":"ContainerStarted","Data":"b1bc003adba6e70ee97b1466a631887ae342b6ebad77469d5707efa48ed0139c"} Nov 03 21:08:19.004042 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:19.003656 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" event={"ID":"7ec5039b-2ca6-4099-9919-12f4b42af9c7","Type":"ContainerStarted","Data":"78c9f1cef8b2dc9360f038aafd810923d4874af421b7e9b607d47ee67bb30aca"} Nov 03 21:08:19.070193 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:19.070160 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:19.071808 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:19.071782 4493 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 03 21:08:19.071932 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:19.071831 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" podUID="7ec5039b-2ca6-4099-9919-12f4b42af9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 03 21:08:19.125007 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:19.124956 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" podStartSLOduration=2.1249437 podStartE2EDuration="2.1249437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:19.124271504 +0000 UTC m=+2370.682262022" watchObservedRunningTime="2026-11-03 21:08:19.1249437 +0000 UTC m=+2370.682934199" Nov 03 21:08:20.071012 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:20.070977 4493 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 03 21:08:20.071475 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:20.071028 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" podUID="7ec5039b-2ca6-4099-9919-12f4b42af9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 03 21:08:21.070275 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:21.070238 4493 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 03 21:08:21.070491 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:21.070298 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" podUID="7ec5039b-2ca6-4099-9919-12f4b42af9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 03 21:08:21.761161 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:08:21.761119 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:08:22.101545 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:22.101408 4493 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 03 21:08:22.101843 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:22.101812 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" podUID="7ec5039b-2ca6-4099-9919-12f4b42af9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 03 21:08:23.071131 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:23.071086 4493 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 03 21:08:23.071641 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:23.071164 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" podUID="7ec5039b-2ca6-4099-9919-12f4b42af9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 03 21:08:24.070654 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:24.070619 4493 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 03 21:08:24.070832 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:24.070670 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" podUID="7ec5039b-2ca6-4099-9919-12f4b42af9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 03 21:08:25.127827 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:25.127798 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:25.128260 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:25.128240 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:25.129158 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:25.129130 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:08:31.880234 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:31.880150 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb"] Nov 03 21:08:31.880735 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:31.880464 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" podUID="569b73ab-788e-432e-877d-440a81b56965" containerName="istio-proxy" containerID="cri-o://d12c29e6d0c1bdfc3c82aec1e35b4a6f91a66d5d2174cf66311d729aaceace80" gracePeriod=30 Nov 03 21:08:34.031365 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:34.031301 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5"] Nov 03 21:08:34.031835 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:34.031551 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5" podUID="c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e" containerName="maas-api" containerID="cri-o://dd7f4e2fd3c99926656a0b5f55769233008a94b464cb954e61352bd3c7fe6e14" gracePeriod=30 Nov 03 21:08:34.167327 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:34.167282 4493 generic.go:356] "Generic (PLEG): container finished" podID="c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e" containerID="dd7f4e2fd3c99926656a0b5f55769233008a94b464cb954e61352bd3c7fe6e14" exitCode=0 Nov 03 21:08:34.167499 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:34.167356 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5" event={"ID":"c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e","Type":"ContainerDied","Data":"dd7f4e2fd3c99926656a0b5f55769233008a94b464cb954e61352bd3c7fe6e14"} Nov 03 21:08:34.369556 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:34.369534 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5" Nov 03 21:08:34.440025 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:34.439988 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-kube-api-access-2bd5d\" (UniqueName: \"kubernetes.io/projected/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-kube-api-access-2bd5d\") pod \"c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e\" (UID: \"c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e\") " Nov 03 21:08:34.440213 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:34.440072 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-openshift-service-ca\") pod \"c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e\" (UID: \"c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e\") " Nov 03 21:08:34.440213 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:34.440122 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-metrics-certs\") pod \"c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e\" (UID: \"c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e\") " Nov 03 21:08:34.440213 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:34.440204 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-maas-api-tls\") pod \"c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e\" (UID: \"c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e\") " Nov 03 21:08:34.440543 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:34.440515 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-openshift-service-ca" pod "c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e" (UID: "c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:34.442884 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:34.442851 4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-openshift-service-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:34.442997 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:34.442881 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-maas-api-tls" pod "c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e" (UID: "c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:08:34.442997 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:34.442931 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-kube-api-access-2bd5d" pod "c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e" (UID: "c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e"). InnerVolumeSpecName "kube-api-access-2bd5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:34.444037 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:34.444014 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-metrics-certs" pod "c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e" (UID: "c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:08:34.545178 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:34.545142 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bd5d\" (UniqueName: \"kubernetes.io/projected/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-kube-api-access-2bd5d\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:34.545447 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:34.545183 4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-metrics-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:34.545447 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:34.545200 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e-maas-api-tls\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:35.218346 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:35.218297 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5" event={"ID":"c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e","Type":"ContainerDied","Data":"415f5aa59f5d97cc15f2dfd7210457e76503c842fee4dc79767a64dc5972c344"} Nov 03 21:08:35.218799 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:35.218398 4493 scope.go:122] "RemoveContainer" containerID="dd7f4e2fd3c99926656a0b5f55769233008a94b464cb954e61352bd3c7fe6e14" Nov 03 21:08:35.218799 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:35.218557 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5" Nov 03 21:08:35.389714 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:35.389679 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5"] Nov 03 21:08:35.441663 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:35.441626 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-88bc35-f6788b5cd-x4px5"] Nov 03 21:08:35.868102 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:35.868065 4493 patch_prober.go:29] interesting pod/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:08:35.868279 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:35.868126 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" podUID="569b73ab-788e-432e-877d-440a81b56965" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:08:36.562560 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:36.562526 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e" path="/var/lib/kubelet/pods/c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e/volumes" Nov 03 21:08:36.708602 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:36.708569 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm"] Nov 03 21:08:36.818051 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:36.817972 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c"] Nov 03 21:08:37.199875 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.199537 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:08:37.231540 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.231512 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-istio-data\") pod \"569b73ab-788e-432e-877d-440a81b56965\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " Nov 03 21:08:37.231725 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.231601 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569b73ab-788e-432e-877d-440a81b56965-istio-token\" (UniqueName: \"kubernetes.io/projected/569b73ab-788e-432e-877d-440a81b56965-istio-token\") pod \"569b73ab-788e-432e-877d-440a81b56965\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " Nov 03 21:08:37.231725 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.231642 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/569b73ab-788e-432e-877d-440a81b56965-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569b73ab-788e-432e-877d-440a81b56965-istiod-ca-cert\") pod \"569b73ab-788e-432e-877d-440a81b56965\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " Nov 03 21:08:37.231725 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.231670 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-workload-certs\") pod \"569b73ab-788e-432e-877d-440a81b56965\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " Nov 03 21:08:37.231725 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.231711 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569b73ab-788e-432e-877d-440a81b56965-kube-api-access-trdk2\" (UniqueName: \"kubernetes.io/projected/569b73ab-788e-432e-877d-440a81b56965-kube-api-access-trdk2\") pod \"569b73ab-788e-432e-877d-440a81b56965\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " Nov 03 21:08:37.231898 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.231815 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-istio-envoy\") pod \"569b73ab-788e-432e-877d-440a81b56965\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " Nov 03 21:08:37.231898 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.231825 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-istio-data" pod "569b73ab-788e-432e-877d-440a81b56965" (UID: "569b73ab-788e-432e-877d-440a81b56965"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:37.232005 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.231917 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/569b73ab-788e-432e-877d-440a81b56965-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569b73ab-788e-432e-877d-440a81b56965-istio-podinfo\") pod \"569b73ab-788e-432e-877d-440a81b56965\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " Nov 03 21:08:37.232005 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.231944 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-workload-socket\") pod \"569b73ab-788e-432e-877d-440a81b56965\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " Nov 03 21:08:37.232005 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.231982 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-credential-socket\") pod \"569b73ab-788e-432e-877d-440a81b56965\" (UID: \"569b73ab-788e-432e-877d-440a81b56965\") " Nov 03 21:08:37.232171 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.232001 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-workload-certs" pod "569b73ab-788e-432e-877d-440a81b56965" (UID: "569b73ab-788e-432e-877d-440a81b56965"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:37.232171 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.232074 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569b73ab-788e-432e-877d-440a81b56965-istiod-ca-cert" pod "569b73ab-788e-432e-877d-440a81b56965" (UID: "569b73ab-788e-432e-877d-440a81b56965"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:37.232302 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.232279 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-credential-socket" pod "569b73ab-788e-432e-877d-440a81b56965" (UID: "569b73ab-788e-432e-877d-440a81b56965"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:37.232431 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.232372 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-workload-socket" pod "569b73ab-788e-432e-877d-440a81b56965" (UID: "569b73ab-788e-432e-877d-440a81b56965"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:37.232937 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.232910 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:37.233056 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.232939 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569b73ab-788e-432e-877d-440a81b56965-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:37.233056 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.232954 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:37.233056 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.232970 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:37.233056 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.232983 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:37.234462 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.234426 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/569b73ab-788e-432e-877d-440a81b56965-istio-podinfo" pod "569b73ab-788e-432e-877d-440a81b56965" (UID: "569b73ab-788e-432e-877d-440a81b56965"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:08:37.234667 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.234529 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569b73ab-788e-432e-877d-440a81b56965-kube-api-access-trdk2" pod "569b73ab-788e-432e-877d-440a81b56965" (UID: "569b73ab-788e-432e-877d-440a81b56965"). InnerVolumeSpecName "kube-api-access-trdk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:37.234667 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.234596 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569b73ab-788e-432e-877d-440a81b56965-istio-token" pod "569b73ab-788e-432e-877d-440a81b56965" (UID: "569b73ab-788e-432e-877d-440a81b56965"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:37.234667 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.234616 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-istio-envoy" pod "569b73ab-788e-432e-877d-440a81b56965" (UID: "569b73ab-788e-432e-877d-440a81b56965"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:37.241089 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.241061 4493 generic.go:356] "Generic (PLEG): container finished" podID="569b73ab-788e-432e-877d-440a81b56965" containerID="d12c29e6d0c1bdfc3c82aec1e35b4a6f91a66d5d2174cf66311d729aaceace80" exitCode=0 Nov 03 21:08:37.241191 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.241115 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" event={"ID":"569b73ab-788e-432e-877d-440a81b56965","Type":"ContainerDied","Data":"d12c29e6d0c1bdfc3c82aec1e35b4a6f91a66d5d2174cf66311d729aaceace80"} Nov 03 21:08:37.241191 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.241170 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" event={"ID":"569b73ab-788e-432e-877d-440a81b56965","Type":"ContainerDied","Data":"216e22e2bbb8e001e3e344622b4eb77ea9cf743574ea3f154bb9ef1f64cdc10c"} Nov 03 21:08:37.241191 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.241184 4493 scope.go:122] "RemoveContainer" containerID="d12c29e6d0c1bdfc3c82aec1e35b4a6f91a66d5d2174cf66311d729aaceace80" Nov 03 21:08:37.241191 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.241171 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb" Nov 03 21:08:37.277080 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.277053 4493 scope.go:122] "RemoveContainer" containerID="703b96b868da99fdea69d97e11417864b67982ec20823ee0f42978b32279c9d0" Nov 03 21:08:37.290622 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.290598 4493 scope.go:122] "RemoveContainer" containerID="d12c29e6d0c1bdfc3c82aec1e35b4a6f91a66d5d2174cf66311d729aaceace80" Nov 03 21:08:37.291243 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:08:37.291214 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d12c29e6d0c1bdfc3c82aec1e35b4a6f91a66d5d2174cf66311d729aaceace80\": container with ID starting with d12c29e6d0c1bdfc3c82aec1e35b4a6f91a66d5d2174cf66311d729aaceace80 not found: ID does not exist" containerID="d12c29e6d0c1bdfc3c82aec1e35b4a6f91a66d5d2174cf66311d729aaceace80" Nov 03 21:08:37.291450 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.291256 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12c29e6d0c1bdfc3c82aec1e35b4a6f91a66d5d2174cf66311d729aaceace80"} err="failed to get container status \"d12c29e6d0c1bdfc3c82aec1e35b4a6f91a66d5d2174cf66311d729aaceace80\": rpc error: code = NotFound desc = could not find container \"d12c29e6d0c1bdfc3c82aec1e35b4a6f91a66d5d2174cf66311d729aaceace80\": container with ID starting with d12c29e6d0c1bdfc3c82aec1e35b4a6f91a66d5d2174cf66311d729aaceace80 not found: ID does not exist" Nov 03 21:08:37.291450 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.291284 4493 scope.go:122] "RemoveContainer" containerID="703b96b868da99fdea69d97e11417864b67982ec20823ee0f42978b32279c9d0" Nov 03 21:08:37.292016 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:08:37.291741 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"703b96b868da99fdea69d97e11417864b67982ec20823ee0f42978b32279c9d0\": container with ID starting with 703b96b868da99fdea69d97e11417864b67982ec20823ee0f42978b32279c9d0 not found: ID does not exist" containerID="703b96b868da99fdea69d97e11417864b67982ec20823ee0f42978b32279c9d0" Nov 03 21:08:37.292016 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.291771 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"703b96b868da99fdea69d97e11417864b67982ec20823ee0f42978b32279c9d0"} err="failed to get container status \"703b96b868da99fdea69d97e11417864b67982ec20823ee0f42978b32279c9d0\": rpc error: code = NotFound desc = could not find container \"703b96b868da99fdea69d97e11417864b67982ec20823ee0f42978b32279c9d0\": container with ID starting with 703b96b868da99fdea69d97e11417864b67982ec20823ee0f42978b32279c9d0 not found: ID does not exist" Nov 03 21:08:37.334527 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.334492 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569b73ab-788e-432e-877d-440a81b56965-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:37.334706 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.334530 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trdk2\" (UniqueName: \"kubernetes.io/projected/569b73ab-788e-432e-877d-440a81b56965-kube-api-access-trdk2\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:37.334706 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.334547 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569b73ab-788e-432e-877d-440a81b56965-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:37.334706 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.334561 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569b73ab-788e-432e-877d-440a81b56965-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:37.437077 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.436998 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb"] Nov 03 21:08:37.440044 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:37.440022 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-46ca0b-openshift-default-6cf5cf5f85-rk6wb"] Nov 03 21:08:38.564082 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:38.564041 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="569b73ab-788e-432e-877d-440a81b56965" path="/var/lib/kubelet/pods/569b73ab-788e-432e-877d-440a81b56965/volumes" Nov 03 21:08:41.033208 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.033171 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858"] Nov 03 21:08:41.034102 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.034081 4493 cpu_manager.go:414] "containerMap: removing container" podUID="569b73ab-788e-432e-877d-440a81b56965" containerName="istio-proxy" Nov 03 21:08:41.034102 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.034102 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569b73ab-788e-432e-877d-440a81b56965" containerName="istio-proxy" Nov 03 21:08:41.034405 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.034120 4493 cpu_manager.go:414] "containerMap: removing container" podUID="569b73ab-788e-432e-877d-440a81b56965" containerName="istio-proxy" Nov 03 21:08:41.034405 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.034128 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569b73ab-788e-432e-877d-440a81b56965" containerName="istio-proxy" Nov 03 21:08:41.034405 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.034149 4493 cpu_manager.go:414] "containerMap: removing container" podUID="c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e" containerName="maas-api" Nov 03 21:08:41.034405 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.034154 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e" containerName="maas-api" Nov 03 21:08:41.034405 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.034277 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="569b73ab-788e-432e-877d-440a81b56965" containerName="istio-proxy" Nov 03 21:08:41.034405 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.034288 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="569b73ab-788e-432e-877d-440a81b56965" containerName="istio-proxy" Nov 03 21:08:41.034405 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.034299 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c951b9fe-0a9e-443c-8b3c-d2a2df7fb91e" containerName="maas-api" Nov 03 21:08:41.039946 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.039924 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858" Nov 03 21:08:41.091656 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.091610 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858"] Nov 03 21:08:41.129924 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.129891 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a905b64-861c-4a3b-94d3-89059db37170-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858\" (UID: \"0a905b64-861c-4a3b-94d3-89059db37170\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858" Nov 03 21:08:41.130132 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.129942 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8zvb\" (UniqueName: \"kubernetes.io/projected/0a905b64-861c-4a3b-94d3-89059db37170-kube-api-access-p8zvb\") pod \"payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858\" (UID: \"0a905b64-861c-4a3b-94d3-89059db37170\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858" Nov 03 21:08:41.156357 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.138476 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-98d1d2-dockercfg-vxc64\"" Nov 03 21:08:41.156357 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.138644 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-98d1d2\"" Nov 03 21:08:41.213426 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.213381 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9"] Nov 03 21:08:41.230004 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.229977 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9" Nov 03 21:08:41.232534 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.232503 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a905b64-861c-4a3b-94d3-89059db37170-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858\" (UID: \"0a905b64-861c-4a3b-94d3-89059db37170\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858" Nov 03 21:08:41.233697 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.233586 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a905b64-861c-4a3b-94d3-89059db37170-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858\" (UID: \"0a905b64-861c-4a3b-94d3-89059db37170\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858" Nov 03 21:08:41.233919 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.233899 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8zvb\" (UniqueName: \"kubernetes.io/projected/0a905b64-861c-4a3b-94d3-89059db37170-kube-api-access-p8zvb\") pod \"payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858\" (UID: \"0a905b64-861c-4a3b-94d3-89059db37170\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858" Nov 03 21:08:41.264322 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.263931 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9"] Nov 03 21:08:41.295441 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.294220 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8zvb\" (UniqueName: \"kubernetes.io/projected/0a905b64-861c-4a3b-94d3-89059db37170-kube-api-access-p8zvb\") pod \"payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858\" (UID: \"0a905b64-861c-4a3b-94d3-89059db37170\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858" Nov 03 21:08:41.335410 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.335376 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956b284c-64a3-42f6-9fb6-20fd8b9f3d24-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9\" (UID: \"956b284c-64a3-42f6-9fb6-20fd8b9f3d24\") " pod="openshift-ingress/payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9" Nov 03 21:08:41.335648 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.335620 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpwtt\" (UniqueName: \"kubernetes.io/projected/956b284c-64a3-42f6-9fb6-20fd8b9f3d24-kube-api-access-fpwtt\") pod \"payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9\" (UID: \"956b284c-64a3-42f6-9fb6-20fd8b9f3d24\") " pod="openshift-ingress/payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9" Nov 03 21:08:41.356771 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.356750 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858" Nov 03 21:08:41.437973 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.437305 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpwtt\" (UniqueName: \"kubernetes.io/projected/956b284c-64a3-42f6-9fb6-20fd8b9f3d24-kube-api-access-fpwtt\") pod \"payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9\" (UID: \"956b284c-64a3-42f6-9fb6-20fd8b9f3d24\") " pod="openshift-ingress/payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9" Nov 03 21:08:41.437973 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.437429 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956b284c-64a3-42f6-9fb6-20fd8b9f3d24-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9\" (UID: \"956b284c-64a3-42f6-9fb6-20fd8b9f3d24\") " pod="openshift-ingress/payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9" Nov 03 21:08:41.438608 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.438589 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956b284c-64a3-42f6-9fb6-20fd8b9f3d24-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9\" (UID: \"956b284c-64a3-42f6-9fb6-20fd8b9f3d24\") " pod="openshift-ingress/payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9" Nov 03 21:08:41.496523 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.496497 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpwtt\" (UniqueName: \"kubernetes.io/projected/956b284c-64a3-42f6-9fb6-20fd8b9f3d24-kube-api-access-fpwtt\") pod \"payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9\" (UID: \"956b284c-64a3-42f6-9fb6-20fd8b9f3d24\") " pod="openshift-ingress/payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9" Nov 03 21:08:41.560918 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.560836 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9" Nov 03 21:08:41.715425 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.715176 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm" podUID="ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd" containerName="payload-processing" containerID="cri-o://716870a9dda8f6d264e262241fac05041ff005cc120aff0237f4a4de00e48028" gracePeriod=25 Nov 03 21:08:41.740189 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.739663 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858"] Nov 03 21:08:41.777913 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:08:41.777870 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a905b64_861c_4a3b_94d3_89059db37170.slice/crio-8c9981c20d4cd76d39312bf065bd097886cdbbd3ef9fdcc5befec9d3b0215837 WatchSource:0}: Error finding container 8c9981c20d4cd76d39312bf065bd097886cdbbd3ef9fdcc5befec9d3b0215837: Status 404 returned error can't find the container with id 8c9981c20d4cd76d39312bf065bd097886cdbbd3ef9fdcc5befec9d3b0215837 Nov 03 21:08:41.818862 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.818725 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c" podUID="cb648add-5f46-4562-83f2-eb0703c98d55" containerName="payload-pre-processing" containerID="cri-o://410cdca36a940890def1a6fa110b240364d82357c14f5543b04047b61ad914e0" gracePeriod=25 Nov 03 21:08:41.947453 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:41.947426 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9"] Nov 03 21:08:42.115486 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.115456 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c" Nov 03 21:08:42.139365 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.139317 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm" Nov 03 21:08:42.154754 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.154735 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb648add-5f46-4562-83f2-eb0703c98d55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb648add-5f46-4562-83f2-eb0703c98d55-plugins-config-volume\") pod \"cb648add-5f46-4562-83f2-eb0703c98d55\" (UID: \"cb648add-5f46-4562-83f2-eb0703c98d55\") " Nov 03 21:08:42.154871 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.154775 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb648add-5f46-4562-83f2-eb0703c98d55-kube-api-access-wb6sn\" (UniqueName: \"kubernetes.io/projected/cb648add-5f46-4562-83f2-eb0703c98d55-kube-api-access-wb6sn\") pod \"cb648add-5f46-4562-83f2-eb0703c98d55\" (UID: \"cb648add-5f46-4562-83f2-eb0703c98d55\") " Nov 03 21:08:42.155113 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.155093 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb648add-5f46-4562-83f2-eb0703c98d55-plugins-config-volume" pod "cb648add-5f46-4562-83f2-eb0703c98d55" (UID: "cb648add-5f46-4562-83f2-eb0703c98d55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:42.155320 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.155307 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb648add-5f46-4562-83f2-eb0703c98d55-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:42.156972 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.156953 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb648add-5f46-4562-83f2-eb0703c98d55-kube-api-access-wb6sn" pod "cb648add-5f46-4562-83f2-eb0703c98d55" (UID: "cb648add-5f46-4562-83f2-eb0703c98d55"). InnerVolumeSpecName "kube-api-access-wb6sn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:42.256649 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.256621 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd-plugins-config-volume\") pod \"ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd\" (UID: \"ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd\") " Nov 03 21:08:42.256810 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.256795 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd-kube-api-access-jwnb5\" (UniqueName: \"kubernetes.io/projected/ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd-kube-api-access-jwnb5\") pod \"ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd\" (UID: \"ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd\") " Nov 03 21:08:42.256999 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.256977 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd-plugins-config-volume" pod "ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd" (UID: "ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:42.257341 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.257308 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb6sn\" (UniqueName: \"kubernetes.io/projected/cb648add-5f46-4562-83f2-eb0703c98d55-kube-api-access-wb6sn\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:42.257391 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.257354 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:42.259215 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.259194 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd-kube-api-access-jwnb5" pod "ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd" (UID: "ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd"). InnerVolumeSpecName "kube-api-access-jwnb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:42.301305 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.301280 4493 generic.go:356] "Generic (PLEG): container finished" podID="cb648add-5f46-4562-83f2-eb0703c98d55" containerID="410cdca36a940890def1a6fa110b240364d82357c14f5543b04047b61ad914e0" exitCode=0 Nov 03 21:08:42.301461 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.301325 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c" Nov 03 21:08:42.301461 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.301378 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c" event={"ID":"cb648add-5f46-4562-83f2-eb0703c98d55","Type":"ContainerDied","Data":"410cdca36a940890def1a6fa110b240364d82357c14f5543b04047b61ad914e0"} Nov 03 21:08:42.301461 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.301432 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c" event={"ID":"cb648add-5f46-4562-83f2-eb0703c98d55","Type":"ContainerDied","Data":"3796b046ed7df7fda668b0c659c87b6f1e5ce66e941a5fa165752bfbfd290a15"} Nov 03 21:08:42.301461 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.301458 4493 scope.go:122] "RemoveContainer" containerID="410cdca36a940890def1a6fa110b240364d82357c14f5543b04047b61ad914e0" Nov 03 21:08:42.303702 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.303673 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858" event={"ID":"0a905b64-861c-4a3b-94d3-89059db37170","Type":"ContainerStarted","Data":"2a9c03c3f5e2b8d3fc7e4a09e762e884e1fb669e701108a0784f9ce6f4cacc05"} Nov 03 21:08:42.303831 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.303723 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858" event={"ID":"0a905b64-861c-4a3b-94d3-89059db37170","Type":"ContainerStarted","Data":"8c9981c20d4cd76d39312bf065bd097886cdbbd3ef9fdcc5befec9d3b0215837"} Nov 03 21:08:42.303831 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.303807 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858" Nov 03 21:08:42.305425 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.305400 4493 generic.go:356] "Generic (PLEG): container finished" podID="ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd" containerID="716870a9dda8f6d264e262241fac05041ff005cc120aff0237f4a4de00e48028" exitCode=0 Nov 03 21:08:42.305538 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.305453 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm" Nov 03 21:08:42.305538 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.305485 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm" event={"ID":"ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd","Type":"ContainerDied","Data":"716870a9dda8f6d264e262241fac05041ff005cc120aff0237f4a4de00e48028"} Nov 03 21:08:42.305659 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.305544 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm" event={"ID":"ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd","Type":"ContainerDied","Data":"41c2a4c551c2d8799cff4828514da298fbda82948b10ca8d872025162d34eddb"} Nov 03 21:08:42.307449 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.307420 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9" event={"ID":"956b284c-64a3-42f6-9fb6-20fd8b9f3d24","Type":"ContainerStarted","Data":"6c7ef55fb3ea39691ec2572a7773d8b7a55f23ddda76a053a8ba4ad60c755a4f"} Nov 03 21:08:42.307542 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.307469 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9" event={"ID":"956b284c-64a3-42f6-9fb6-20fd8b9f3d24","Type":"ContainerStarted","Data":"457f06cd3c41903c1e61624352d95062a44538b25316b7f96f630edf3dedd6c7"} Nov 03 21:08:42.307542 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.307510 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9" Nov 03 21:08:42.325316 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.325300 4493 scope.go:122] "RemoveContainer" containerID="410cdca36a940890def1a6fa110b240364d82357c14f5543b04047b61ad914e0" Nov 03 21:08:42.325605 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:08:42.325587 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"410cdca36a940890def1a6fa110b240364d82357c14f5543b04047b61ad914e0\": container with ID starting with 410cdca36a940890def1a6fa110b240364d82357c14f5543b04047b61ad914e0 not found: ID does not exist" containerID="410cdca36a940890def1a6fa110b240364d82357c14f5543b04047b61ad914e0" Nov 03 21:08:42.325650 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.325613 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"410cdca36a940890def1a6fa110b240364d82357c14f5543b04047b61ad914e0"} err="failed to get container status \"410cdca36a940890def1a6fa110b240364d82357c14f5543b04047b61ad914e0\": rpc error: code = NotFound desc = could not find container \"410cdca36a940890def1a6fa110b240364d82357c14f5543b04047b61ad914e0\": container with ID starting with 410cdca36a940890def1a6fa110b240364d82357c14f5543b04047b61ad914e0 not found: ID does not exist" Nov 03 21:08:42.325650 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.325630 4493 scope.go:122] "RemoveContainer" containerID="716870a9dda8f6d264e262241fac05041ff005cc120aff0237f4a4de00e48028" Nov 03 21:08:42.335310 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.335288 4493 scope.go:122] "RemoveContainer" containerID="716870a9dda8f6d264e262241fac05041ff005cc120aff0237f4a4de00e48028" Nov 03 21:08:42.335660 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:08:42.335641 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"716870a9dda8f6d264e262241fac05041ff005cc120aff0237f4a4de00e48028\": container with ID starting with 716870a9dda8f6d264e262241fac05041ff005cc120aff0237f4a4de00e48028 not found: ID does not exist" containerID="716870a9dda8f6d264e262241fac05041ff005cc120aff0237f4a4de00e48028" Nov 03 21:08:42.335714 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.335665 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716870a9dda8f6d264e262241fac05041ff005cc120aff0237f4a4de00e48028"} err="failed to get container status \"716870a9dda8f6d264e262241fac05041ff005cc120aff0237f4a4de00e48028\": rpc error: code = NotFound desc = could not find container \"716870a9dda8f6d264e262241fac05041ff005cc120aff0237f4a4de00e48028\": container with ID starting with 716870a9dda8f6d264e262241fac05041ff005cc120aff0237f4a4de00e48028 not found: ID does not exist" Nov 03 21:08:42.358713 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.358654 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwnb5\" (UniqueName: \"kubernetes.io/projected/ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd-kube-api-access-jwnb5\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:42.469560 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.469522 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm"] Nov 03 21:08:42.472607 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.472580 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-kcgdm"] Nov 03 21:08:42.563922 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.563886 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd" path="/var/lib/kubelet/pods/ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd/volumes" Nov 03 21:08:42.584245 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.584212 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c"] Nov 03 21:08:42.587246 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.587227 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-pxl4c"] Nov 03 21:08:42.700347 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.700220 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9" podStartSLOduration=1.700203948 podStartE2EDuration="1.700203948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:42.698819369 +0000 UTC m=+2394.256809901" watchObservedRunningTime="2026-11-03 21:08:42.700203948 +0000 UTC m=+2394.258194446" Nov 03 21:08:42.818772 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:42.818710 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858" podStartSLOduration=2.818689992 podStartE2EDuration="2.818689992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:42.817817716 +0000 UTC m=+2394.375808216" watchObservedRunningTime="2026-11-03 21:08:42.818689992 +0000 UTC m=+2394.376680492" Nov 03 21:08:43.455561 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:43.455526 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw"] Nov 03 21:08:43.456090 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:43.455782 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" podUID="8c88cfbc-20ad-47cb-8138-cabbaf57eb1d" containerName="istio-proxy" containerID="cri-o://0910a8d7214023a76a1f97090bca3f3684262b9f288d63add93c6b5b39f8ed95" gracePeriod=30 Nov 03 21:08:44.562346 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:44.562307 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb648add-5f46-4562-83f2-eb0703c98d55" path="/var/lib/kubelet/pods/cb648add-5f46-4562-83f2-eb0703c98d55/volumes" Nov 03 21:08:45.367229 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:45.367194 4493 patch_prober.go:29] interesting pod/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:08:45.367471 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:45.367257 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" podUID="8c88cfbc-20ad-47cb-8138-cabbaf57eb1d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:08:48.739388 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.739304 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 21:08:48.809934 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.809910 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-workload-certs\") pod \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " Nov 03 21:08:48.810050 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.810007 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-workload-socket\") pod \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " Nov 03 21:08:48.810050 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.810044 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-token\" (UniqueName: \"kubernetes.io/projected/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-token\") pod \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " Nov 03 21:08:48.810132 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.810077 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-envoy\") pod \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " Nov 03 21:08:48.810132 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.810106 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-credential-socket\") pod \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " Nov 03 21:08:48.810208 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.810159 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-workload-certs" pod "8c88cfbc-20ad-47cb-8138-cabbaf57eb1d" (UID: "8c88cfbc-20ad-47cb-8138-cabbaf57eb1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:48.810208 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.810171 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istiod-ca-cert\") pod \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " Nov 03 21:08:48.810313 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.810290 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-workload-socket" pod "8c88cfbc-20ad-47cb-8138-cabbaf57eb1d" (UID: "8c88cfbc-20ad-47cb-8138-cabbaf57eb1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:48.810434 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.810414 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-credential-socket" pod "8c88cfbc-20ad-47cb-8138-cabbaf57eb1d" (UID: "8c88cfbc-20ad-47cb-8138-cabbaf57eb1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:48.810434 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.810424 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-kube-api-access-9ccb4\" (UniqueName: \"kubernetes.io/projected/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-kube-api-access-9ccb4\") pod \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " Nov 03 21:08:48.810564 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.810463 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-podinfo\") pod \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " Nov 03 21:08:48.810564 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.810511 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istiod-ca-cert" pod "8c88cfbc-20ad-47cb-8138-cabbaf57eb1d" (UID: "8c88cfbc-20ad-47cb-8138-cabbaf57eb1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:48.810564 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.810523 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-data\") pod \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\" (UID: \"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d\") " Nov 03 21:08:48.810724 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.810682 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-data" pod "8c88cfbc-20ad-47cb-8138-cabbaf57eb1d" (UID: "8c88cfbc-20ad-47cb-8138-cabbaf57eb1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:48.811467 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.811442 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:48.811579 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.811470 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:48.811579 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.811485 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:48.811579 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.811500 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:48.811579 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.811514 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:48.812401 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.812370 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-envoy" pod "8c88cfbc-20ad-47cb-8138-cabbaf57eb1d" (UID: "8c88cfbc-20ad-47cb-8138-cabbaf57eb1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:48.812608 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.812586 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-token" pod "8c88cfbc-20ad-47cb-8138-cabbaf57eb1d" (UID: "8c88cfbc-20ad-47cb-8138-cabbaf57eb1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:48.812721 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.812618 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-kube-api-access-9ccb4" pod "8c88cfbc-20ad-47cb-8138-cabbaf57eb1d" (UID: "8c88cfbc-20ad-47cb-8138-cabbaf57eb1d"). InnerVolumeSpecName "kube-api-access-9ccb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:48.812770 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.812758 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-podinfo" pod "8c88cfbc-20ad-47cb-8138-cabbaf57eb1d" (UID: "8c88cfbc-20ad-47cb-8138-cabbaf57eb1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:08:48.912701 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.912670 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ccb4\" (UniqueName: \"kubernetes.io/projected/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-kube-api-access-9ccb4\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:48.912701 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.912701 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:48.912701 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.912711 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:48.912927 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:48.912719 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:49.024736 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:49.024659 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 21:08:49.025321 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:49.025302 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 21:08:49.359385 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:49.359325 4493 generic.go:356] "Generic (PLEG): container finished" podID="8c88cfbc-20ad-47cb-8138-cabbaf57eb1d" containerID="0910a8d7214023a76a1f97090bca3f3684262b9f288d63add93c6b5b39f8ed95" exitCode=0 Nov 03 21:08:49.359385 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:49.359366 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" event={"ID":"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d","Type":"ContainerDied","Data":"0910a8d7214023a76a1f97090bca3f3684262b9f288d63add93c6b5b39f8ed95"} Nov 03 21:08:49.359571 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:49.359419 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" Nov 03 21:08:49.359571 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:49.359464 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw" event={"ID":"8c88cfbc-20ad-47cb-8138-cabbaf57eb1d","Type":"ContainerDied","Data":"bcc6b99a1be7ab72275815bb59ff648d5822e4ca6eadbb2548c2e640352a66d4"} Nov 03 21:08:49.359571 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:49.359500 4493 scope.go:122] "RemoveContainer" containerID="0910a8d7214023a76a1f97090bca3f3684262b9f288d63add93c6b5b39f8ed95" Nov 03 21:08:49.372738 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:49.372716 4493 scope.go:122] "RemoveContainer" containerID="398ad7ddae6b1b89e09731f3562f8648ed7f2a0e9e2abbb4158b4db0369da269" Nov 03 21:08:49.383910 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:49.383886 4493 scope.go:122] "RemoveContainer" containerID="0910a8d7214023a76a1f97090bca3f3684262b9f288d63add93c6b5b39f8ed95" Nov 03 21:08:49.384193 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:08:49.384164 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0910a8d7214023a76a1f97090bca3f3684262b9f288d63add93c6b5b39f8ed95\": container with ID starting with 0910a8d7214023a76a1f97090bca3f3684262b9f288d63add93c6b5b39f8ed95 not found: ID does not exist" containerID="0910a8d7214023a76a1f97090bca3f3684262b9f288d63add93c6b5b39f8ed95" Nov 03 21:08:49.384258 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:49.384202 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0910a8d7214023a76a1f97090bca3f3684262b9f288d63add93c6b5b39f8ed95"} err="failed to get container status \"0910a8d7214023a76a1f97090bca3f3684262b9f288d63add93c6b5b39f8ed95\": rpc error: code = NotFound desc = could not find container \"0910a8d7214023a76a1f97090bca3f3684262b9f288d63add93c6b5b39f8ed95\": container with ID starting with 0910a8d7214023a76a1f97090bca3f3684262b9f288d63add93c6b5b39f8ed95 not found: ID does not exist" Nov 03 21:08:49.384258 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:49.384220 4493 scope.go:122] "RemoveContainer" containerID="398ad7ddae6b1b89e09731f3562f8648ed7f2a0e9e2abbb4158b4db0369da269" Nov 03 21:08:49.384513 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:08:49.384492 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"398ad7ddae6b1b89e09731f3562f8648ed7f2a0e9e2abbb4158b4db0369da269\": container with ID starting with 398ad7ddae6b1b89e09731f3562f8648ed7f2a0e9e2abbb4158b4db0369da269 not found: ID does not exist" containerID="398ad7ddae6b1b89e09731f3562f8648ed7f2a0e9e2abbb4158b4db0369da269" Nov 03 21:08:49.384566 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:49.384519 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"398ad7ddae6b1b89e09731f3562f8648ed7f2a0e9e2abbb4158b4db0369da269"} err="failed to get container status \"398ad7ddae6b1b89e09731f3562f8648ed7f2a0e9e2abbb4158b4db0369da269\": rpc error: code = NotFound desc = could not find container \"398ad7ddae6b1b89e09731f3562f8648ed7f2a0e9e2abbb4158b4db0369da269\": container with ID starting with 398ad7ddae6b1b89e09731f3562f8648ed7f2a0e9e2abbb4158b4db0369da269 not found: ID does not exist" Nov 03 21:08:49.530550 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:49.530518 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw"] Nov 03 21:08:49.533596 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:49.533568 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-88bc35-openshift-default-86cc4ff85f-tk5gw"] Nov 03 21:08:50.562181 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:50.562149 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c88cfbc-20ad-47cb-8138-cabbaf57eb1d" path="/var/lib/kubelet/pods/8c88cfbc-20ad-47cb-8138-cabbaf57eb1d/volumes" Nov 03 21:08:51.745300 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:08:51.745268 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:08:53.320589 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:53.320560 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9" Nov 03 21:08:53.321047 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:53.320651 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858" Nov 03 21:08:57.029791 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:57.029760 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn"] Nov 03 21:08:57.043861 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:57.043826 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6"] Nov 03 21:08:57.080998 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:08:57.080970 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m"] Nov 03 21:09:10.493170 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:10.493138 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d"] Nov 03 21:09:12.052863 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.052822 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" containerID="cri-o://9a67360724ee710140e42327eb5c36a330ea3c6898dde54aebbe5f9f70ab2618" gracePeriod=45 Nov 03 21:09:12.064746 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.064714 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" containerID="cri-o://22ae47865f30cdb42e4034f579e425d31c4d39f0ffa80a287fa0a00cb992850b" gracePeriod=45 Nov 03 21:09:12.101322 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.101294 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" containerID="cri-o://4d37cb51ad870f0a92b70db7f89a6ce8a8c5d9fd32eb7dff92a982cd6add0089" gracePeriod=45 Nov 03 21:09:12.110414 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.110387 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": dial tcp 10.133.0.50:8000: connect: connection refused" Nov 03 21:09:12.392907 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.392880 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 21:09:12.400893 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.400866 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 21:09:12.406405 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.406378 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 21:09:12.473527 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.473496 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-home\" (UniqueName: \"kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-home\") pod \"46e22045-16c7-48e6-9046-87433500fe27\" (UID: \"46e22045-16c7-48e6-9046-87433500fe27\") " Nov 03 21:09:12.473704 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.473576 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-model-cache\") pod \"827fd825-7e75-4174-b522-2de39a8cce28\" (UID: \"827fd825-7e75-4174-b522-2de39a8cce28\") " Nov 03 21:09:12.473704 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.473602 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46e22045-16c7-48e6-9046-87433500fe27-tls-certs\" (UniqueName: \"kubernetes.io/secret/46e22045-16c7-48e6-9046-87433500fe27-tls-certs\") pod \"46e22045-16c7-48e6-9046-87433500fe27\" (UID: \"46e22045-16c7-48e6-9046-87433500fe27\") " Nov 03 21:09:12.473704 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.473619 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-tmp-dir\") pod \"827fd825-7e75-4174-b522-2de39a8cce28\" (UID: \"827fd825-7e75-4174-b522-2de39a8cce28\") " Nov 03 21:09:12.473704 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.473635 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-dshm\") pod \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\" (UID: \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\") " Nov 03 21:09:12.473704 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.473700 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-model-cache\") pod \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\" (UID: \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\") " Nov 03 21:09:12.473981 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.473735 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-tmp-dir\") pod \"46e22045-16c7-48e6-9046-87433500fe27\" (UID: \"46e22045-16c7-48e6-9046-87433500fe27\") " Nov 03 21:09:12.473981 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.473797 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46e22045-16c7-48e6-9046-87433500fe27-kube-api-access-mg54k\" (UniqueName: \"kubernetes.io/projected/46e22045-16c7-48e6-9046-87433500fe27-kube-api-access-mg54k\") pod \"46e22045-16c7-48e6-9046-87433500fe27\" (UID: \"46e22045-16c7-48e6-9046-87433500fe27\") " Nov 03 21:09:12.473981 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.473804 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-home" pod "46e22045-16c7-48e6-9046-87433500fe27" (UID: "46e22045-16c7-48e6-9046-87433500fe27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:12.473981 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.473866 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-tmp-dir" pod "827fd825-7e75-4174-b522-2de39a8cce28" (UID: "827fd825-7e75-4174-b522-2de39a8cce28"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:12.473981 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.473874 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-model-cache" pod "827fd825-7e75-4174-b522-2de39a8cce28" (UID: "827fd825-7e75-4174-b522-2de39a8cce28"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:12.473981 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.473882 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-dshm\" (UniqueName: \"kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-dshm\") pod \"827fd825-7e75-4174-b522-2de39a8cce28\" (UID: \"827fd825-7e75-4174-b522-2de39a8cce28\") " Nov 03 21:09:12.473981 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.473941 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8205aa0-9b8a-4961-bd95-947ac5edfdff-tls-certs\" (UniqueName: \"kubernetes.io/secret/e8205aa0-9b8a-4961-bd95-947ac5edfdff-tls-certs\") pod \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\" (UID: \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\") " Nov 03 21:09:12.474474 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.474016 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/827fd825-7e75-4174-b522-2de39a8cce28-kube-api-access-zc2px\" (UniqueName: \"kubernetes.io/projected/827fd825-7e75-4174-b522-2de39a8cce28-kube-api-access-zc2px\") pod \"827fd825-7e75-4174-b522-2de39a8cce28\" (UID: \"827fd825-7e75-4174-b522-2de39a8cce28\") " Nov 03 21:09:12.474474 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.474092 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-tmp-dir" pod "46e22045-16c7-48e6-9046-87433500fe27" (UID: "46e22045-16c7-48e6-9046-87433500fe27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:12.474474 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.474148 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-model-cache\") pod \"46e22045-16c7-48e6-9046-87433500fe27\" (UID: \"46e22045-16c7-48e6-9046-87433500fe27\") " Nov 03 21:09:12.474474 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.474041 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-model-cache" pod "e8205aa0-9b8a-4961-bd95-947ac5edfdff" (UID: "e8205aa0-9b8a-4961-bd95-947ac5edfdff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:12.474474 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.474399 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-tmp-dir\") pod \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\" (UID: \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\") " Nov 03 21:09:12.474474 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.474410 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-model-cache" pod "46e22045-16c7-48e6-9046-87433500fe27" (UID: "46e22045-16c7-48e6-9046-87433500fe27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:12.474474 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.474452 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-home\" (UniqueName: \"kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-home\") pod \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\" (UID: \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\") " Nov 03 21:09:12.475034 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.474640 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-dshm\" (UniqueName: \"kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-dshm\") pod \"46e22045-16c7-48e6-9046-87433500fe27\" (UID: \"46e22045-16c7-48e6-9046-87433500fe27\") " Nov 03 21:09:12.475034 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.474699 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/827fd825-7e75-4174-b522-2de39a8cce28-tls-certs\" (UniqueName: \"kubernetes.io/secret/827fd825-7e75-4174-b522-2de39a8cce28-tls-certs\") pod \"827fd825-7e75-4174-b522-2de39a8cce28\" (UID: \"827fd825-7e75-4174-b522-2de39a8cce28\") " Nov 03 21:09:12.475034 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.474740 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8205aa0-9b8a-4961-bd95-947ac5edfdff-kube-api-access-6dnxw\" (UniqueName: \"kubernetes.io/projected/e8205aa0-9b8a-4961-bd95-947ac5edfdff-kube-api-access-6dnxw\") pod \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\" (UID: \"e8205aa0-9b8a-4961-bd95-947ac5edfdff\") " Nov 03 21:09:12.475034 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.474801 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-home\" (UniqueName: \"kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-home\") pod \"827fd825-7e75-4174-b522-2de39a8cce28\" (UID: \"827fd825-7e75-4174-b522-2de39a8cce28\") " Nov 03 21:09:12.475514 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.475488 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-home\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:12.475632 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.475517 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-model-cache\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:12.475632 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.475531 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-tmp-dir\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:12.475632 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.475545 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-model-cache\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:12.475632 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.475557 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-tmp-dir\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:12.475632 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.475571 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-model-cache\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:12.476416 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.475814 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-home" pod "827fd825-7e75-4174-b522-2de39a8cce28" (UID: "827fd825-7e75-4174-b522-2de39a8cce28"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:12.476987 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.476960 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-dshm" pod "e8205aa0-9b8a-4961-bd95-947ac5edfdff" (UID: "e8205aa0-9b8a-4961-bd95-947ac5edfdff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:12.477206 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.477176 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-tmp-dir" pod "e8205aa0-9b8a-4961-bd95-947ac5edfdff" (UID: "e8205aa0-9b8a-4961-bd95-947ac5edfdff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:12.477325 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.477253 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-dshm" pod "827fd825-7e75-4174-b522-2de39a8cce28" (UID: "827fd825-7e75-4174-b522-2de39a8cce28"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:12.477428 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.477397 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-home" pod "e8205aa0-9b8a-4961-bd95-947ac5edfdff" (UID: "e8205aa0-9b8a-4961-bd95-947ac5edfdff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:12.477537 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.477510 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/827fd825-7e75-4174-b522-2de39a8cce28-kube-api-access-zc2px" pod "827fd825-7e75-4174-b522-2de39a8cce28" (UID: "827fd825-7e75-4174-b522-2de39a8cce28"). InnerVolumeSpecName "kube-api-access-zc2px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:12.477846 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.477822 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e22045-16c7-48e6-9046-87433500fe27-kube-api-access-mg54k" pod "46e22045-16c7-48e6-9046-87433500fe27" (UID: "46e22045-16c7-48e6-9046-87433500fe27"). InnerVolumeSpecName "kube-api-access-mg54k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:12.478055 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.478033 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-dshm" pod "46e22045-16c7-48e6-9046-87433500fe27" (UID: "46e22045-16c7-48e6-9046-87433500fe27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:12.478523 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.478501 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46e22045-16c7-48e6-9046-87433500fe27-tls-certs" pod "46e22045-16c7-48e6-9046-87433500fe27" (UID: "46e22045-16c7-48e6-9046-87433500fe27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:12.479273 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.479251 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/827fd825-7e75-4174-b522-2de39a8cce28-tls-certs" pod "827fd825-7e75-4174-b522-2de39a8cce28" (UID: "827fd825-7e75-4174-b522-2de39a8cce28"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:12.479484 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.479465 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8205aa0-9b8a-4961-bd95-947ac5edfdff-tls-certs" pod "e8205aa0-9b8a-4961-bd95-947ac5edfdff" (UID: "e8205aa0-9b8a-4961-bd95-947ac5edfdff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:12.479536 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.479514 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8205aa0-9b8a-4961-bd95-947ac5edfdff-kube-api-access-6dnxw" pod "e8205aa0-9b8a-4961-bd95-947ac5edfdff" (UID: "e8205aa0-9b8a-4961-bd95-947ac5edfdff"). InnerVolumeSpecName "kube-api-access-6dnxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:12.531929 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.531890 4493 generic.go:356] "Generic (PLEG): container finished" podID="827fd825-7e75-4174-b522-2de39a8cce28" containerID="22ae47865f30cdb42e4034f579e425d31c4d39f0ffa80a287fa0a00cb992850b" exitCode=0 Nov 03 21:09:12.532115 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.531942 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" Nov 03 21:09:12.532115 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.531972 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" event={"ID":"827fd825-7e75-4174-b522-2de39a8cce28","Type":"ContainerDied","Data":"22ae47865f30cdb42e4034f579e425d31c4d39f0ffa80a287fa0a00cb992850b"} Nov 03 21:09:12.532115 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.532024 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6" event={"ID":"827fd825-7e75-4174-b522-2de39a8cce28","Type":"ContainerDied","Data":"168bc7afee6b19459626c8c2b426e7c81a46f0bed94a31a4e34f669b7f9273cd"} Nov 03 21:09:12.532115 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.532048 4493 scope.go:122] "RemoveContainer" containerID="22ae47865f30cdb42e4034f579e425d31c4d39f0ffa80a287fa0a00cb992850b" Nov 03 21:09:12.533917 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.533888 4493 generic.go:356] "Generic (PLEG): container finished" podID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerID="9a67360724ee710140e42327eb5c36a330ea3c6898dde54aebbe5f9f70ab2618" exitCode=0 Nov 03 21:09:12.534053 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.533977 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" Nov 03 21:09:12.534193 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.534173 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" event={"ID":"e8205aa0-9b8a-4961-bd95-947ac5edfdff","Type":"ContainerDied","Data":"9a67360724ee710140e42327eb5c36a330ea3c6898dde54aebbe5f9f70ab2618"} Nov 03 21:09:12.534256 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.534213 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn" event={"ID":"e8205aa0-9b8a-4961-bd95-947ac5edfdff","Type":"ContainerDied","Data":"804f18dcfa8402a5decfa6d0bf01ffbf3d8edd51bdb220ba44f18b0b2d67d816"} Nov 03 21:09:12.536079 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.536054 4493 generic.go:356] "Generic (PLEG): container finished" podID="46e22045-16c7-48e6-9046-87433500fe27" containerID="4d37cb51ad870f0a92b70db7f89a6ce8a8c5d9fd32eb7dff92a982cd6add0089" exitCode=0 Nov 03 21:09:12.536274 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.536257 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" event={"ID":"46e22045-16c7-48e6-9046-87433500fe27","Type":"ContainerDied","Data":"4d37cb51ad870f0a92b70db7f89a6ce8a8c5d9fd32eb7dff92a982cd6add0089"} Nov 03 21:09:12.536376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.536290 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" event={"ID":"46e22045-16c7-48e6-9046-87433500fe27","Type":"ContainerDied","Data":"fcd5aa8c1c86c1c1aca826b6c5aae5bdc2995afc4f7ccfab2a17eb7283162ce7"} Nov 03 21:09:12.536445 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.536434 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m" Nov 03 21:09:12.546398 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.546319 4493 scope.go:122] "RemoveContainer" containerID="22ae47865f30cdb42e4034f579e425d31c4d39f0ffa80a287fa0a00cb992850b" Nov 03 21:09:12.549068 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:09:12.549041 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22ae47865f30cdb42e4034f579e425d31c4d39f0ffa80a287fa0a00cb992850b\": container with ID starting with 22ae47865f30cdb42e4034f579e425d31c4d39f0ffa80a287fa0a00cb992850b not found: ID does not exist" containerID="22ae47865f30cdb42e4034f579e425d31c4d39f0ffa80a287fa0a00cb992850b" Nov 03 21:09:12.549175 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.549079 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ae47865f30cdb42e4034f579e425d31c4d39f0ffa80a287fa0a00cb992850b"} err="failed to get container status \"22ae47865f30cdb42e4034f579e425d31c4d39f0ffa80a287fa0a00cb992850b\": rpc error: code = NotFound desc = could not find container \"22ae47865f30cdb42e4034f579e425d31c4d39f0ffa80a287fa0a00cb992850b\": container with ID starting with 22ae47865f30cdb42e4034f579e425d31c4d39f0ffa80a287fa0a00cb992850b not found: ID does not exist" Nov 03 21:09:12.549175 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.549104 4493 scope.go:122] "RemoveContainer" containerID="9a67360724ee710140e42327eb5c36a330ea3c6898dde54aebbe5f9f70ab2618" Nov 03 21:09:12.559801 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.559784 4493 scope.go:122] "RemoveContainer" containerID="9a67360724ee710140e42327eb5c36a330ea3c6898dde54aebbe5f9f70ab2618" Nov 03 21:09:12.560095 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:09:12.560072 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a67360724ee710140e42327eb5c36a330ea3c6898dde54aebbe5f9f70ab2618\": container with ID starting with 9a67360724ee710140e42327eb5c36a330ea3c6898dde54aebbe5f9f70ab2618 not found: ID does not exist" containerID="9a67360724ee710140e42327eb5c36a330ea3c6898dde54aebbe5f9f70ab2618" Nov 03 21:09:12.560180 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.560105 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a67360724ee710140e42327eb5c36a330ea3c6898dde54aebbe5f9f70ab2618"} err="failed to get container status \"9a67360724ee710140e42327eb5c36a330ea3c6898dde54aebbe5f9f70ab2618\": rpc error: code = NotFound desc = could not find container \"9a67360724ee710140e42327eb5c36a330ea3c6898dde54aebbe5f9f70ab2618\": container with ID starting with 9a67360724ee710140e42327eb5c36a330ea3c6898dde54aebbe5f9f70ab2618 not found: ID does not exist" Nov 03 21:09:12.560180 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.560130 4493 scope.go:122] "RemoveContainer" containerID="4d37cb51ad870f0a92b70db7f89a6ce8a8c5d9fd32eb7dff92a982cd6add0089" Nov 03 21:09:12.569501 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.569483 4493 scope.go:122] "RemoveContainer" containerID="4d37cb51ad870f0a92b70db7f89a6ce8a8c5d9fd32eb7dff92a982cd6add0089" Nov 03 21:09:12.569757 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:09:12.569739 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d37cb51ad870f0a92b70db7f89a6ce8a8c5d9fd32eb7dff92a982cd6add0089\": container with ID starting with 4d37cb51ad870f0a92b70db7f89a6ce8a8c5d9fd32eb7dff92a982cd6add0089 not found: ID does not exist" containerID="4d37cb51ad870f0a92b70db7f89a6ce8a8c5d9fd32eb7dff92a982cd6add0089" Nov 03 21:09:12.569805 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.569769 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d37cb51ad870f0a92b70db7f89a6ce8a8c5d9fd32eb7dff92a982cd6add0089"} err="failed to get container status \"4d37cb51ad870f0a92b70db7f89a6ce8a8c5d9fd32eb7dff92a982cd6add0089\": rpc error: code = NotFound desc = could not find container \"4d37cb51ad870f0a92b70db7f89a6ce8a8c5d9fd32eb7dff92a982cd6add0089\": container with ID starting with 4d37cb51ad870f0a92b70db7f89a6ce8a8c5d9fd32eb7dff92a982cd6add0089 not found: ID does not exist" Nov 03 21:09:12.576588 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.576567 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46e22045-16c7-48e6-9046-87433500fe27-tls-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:12.576588 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.576590 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-dshm\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:12.576695 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.576599 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg54k\" (UniqueName: \"kubernetes.io/projected/46e22045-16c7-48e6-9046-87433500fe27-kube-api-access-mg54k\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:12.576695 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.576609 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-dshm\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:12.576695 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.576618 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8205aa0-9b8a-4961-bd95-947ac5edfdff-tls-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:12.576695 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.576626 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc2px\" (UniqueName: \"kubernetes.io/projected/827fd825-7e75-4174-b522-2de39a8cce28-kube-api-access-zc2px\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:12.576695 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.576635 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-tmp-dir\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:12.576695 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.576643 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8205aa0-9b8a-4961-bd95-947ac5edfdff-home\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:12.576695 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.576652 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46e22045-16c7-48e6-9046-87433500fe27-dshm\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:12.576695 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.576659 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/827fd825-7e75-4174-b522-2de39a8cce28-tls-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:12.576695 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.576667 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dnxw\" (UniqueName: \"kubernetes.io/projected/e8205aa0-9b8a-4961-bd95-947ac5edfdff-kube-api-access-6dnxw\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:12.576695 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.576675 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/827fd825-7e75-4174-b522-2de39a8cce28-home\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:12.862283 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.861303 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m"] Nov 03 21:09:12.865725 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.865698 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-a19980/simulator-w6-a19980-kserve-586549d789-z999m"] Nov 03 21:09:12.976196 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.976163 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn"] Nov 03 21:09:12.979484 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:12.979465 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-a19980/distinct-2-w6-a19980-kserve-68ccb9958d-9njbn"] Nov 03 21:09:13.091111 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:13.091081 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6"] Nov 03 21:09:13.094087 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:13.094061 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-a19980/premium-w6-a19980-kserve-6664698c8d-bvrq6"] Nov 03 21:09:14.562988 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:14.562955 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46e22045-16c7-48e6-9046-87433500fe27" path="/var/lib/kubelet/pods/46e22045-16c7-48e6-9046-87433500fe27/volumes" Nov 03 21:09:14.563460 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:14.563444 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="827fd825-7e75-4174-b522-2de39a8cce28" path="/var/lib/kubelet/pods/827fd825-7e75-4174-b522-2de39a8cce28/volumes" Nov 03 21:09:14.563866 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:14.563853 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" path="/var/lib/kubelet/pods/e8205aa0-9b8a-4961-bd95-947ac5edfdff/volumes" Nov 03 21:09:21.490462 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:21.490426 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j"] Nov 03 21:09:21.490905 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:21.490621 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j" podUID="bc04227c-da6e-421c-b258-135ebe1e519e" containerName="maas-api" containerID="cri-o://5ed251b06cc3759bea07888a96482fb1abc5fc9f0f78995dcb3ee66a4fd5e710" gracePeriod=30 Nov 03 21:09:21.765224 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:09:21.765132 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:09:21.800092 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:21.800059 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j" Nov 03 21:09:21.814152 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:21.814125 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc04227c-da6e-421c-b258-135ebe1e519e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc04227c-da6e-421c-b258-135ebe1e519e-maas-api-tls\") pod \"bc04227c-da6e-421c-b258-135ebe1e519e\" (UID: \"bc04227c-da6e-421c-b258-135ebe1e519e\") " Nov 03 21:09:21.814306 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:21.814246 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc04227c-da6e-421c-b258-135ebe1e519e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc04227c-da6e-421c-b258-135ebe1e519e-openshift-service-ca\") pod \"bc04227c-da6e-421c-b258-135ebe1e519e\" (UID: \"bc04227c-da6e-421c-b258-135ebe1e519e\") " Nov 03 21:09:21.814306 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:21.814264 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc04227c-da6e-421c-b258-135ebe1e519e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc04227c-da6e-421c-b258-135ebe1e519e-metrics-certs\") pod \"bc04227c-da6e-421c-b258-135ebe1e519e\" (UID: \"bc04227c-da6e-421c-b258-135ebe1e519e\") " Nov 03 21:09:21.814473 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:21.814349 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc04227c-da6e-421c-b258-135ebe1e519e-kube-api-access-8zncw\" (UniqueName: \"kubernetes.io/projected/bc04227c-da6e-421c-b258-135ebe1e519e-kube-api-access-8zncw\") pod \"bc04227c-da6e-421c-b258-135ebe1e519e\" (UID: \"bc04227c-da6e-421c-b258-135ebe1e519e\") " Nov 03 21:09:21.814690 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:21.814657 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc04227c-da6e-421c-b258-135ebe1e519e-openshift-service-ca" pod "bc04227c-da6e-421c-b258-135ebe1e519e" (UID: "bc04227c-da6e-421c-b258-135ebe1e519e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:21.814853 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:21.814837 4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc04227c-da6e-421c-b258-135ebe1e519e-openshift-service-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:21.816822 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:21.816793 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc04227c-da6e-421c-b258-135ebe1e519e-maas-api-tls" pod "bc04227c-da6e-421c-b258-135ebe1e519e" (UID: "bc04227c-da6e-421c-b258-135ebe1e519e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:21.817265 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:21.817238 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc04227c-da6e-421c-b258-135ebe1e519e-metrics-certs" pod "bc04227c-da6e-421c-b258-135ebe1e519e" (UID: "bc04227c-da6e-421c-b258-135ebe1e519e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:21.817423 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:21.817362 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc04227c-da6e-421c-b258-135ebe1e519e-kube-api-access-8zncw" pod "bc04227c-da6e-421c-b258-135ebe1e519e" (UID: "bc04227c-da6e-421c-b258-135ebe1e519e"). InnerVolumeSpecName "kube-api-access-8zncw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:21.916857 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:21.916787 4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc04227c-da6e-421c-b258-135ebe1e519e-metrics-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:21.916857 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:21.916827 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zncw\" (UniqueName: \"kubernetes.io/projected/bc04227c-da6e-421c-b258-135ebe1e519e-kube-api-access-8zncw\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:21.916857 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:21.916847 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc04227c-da6e-421c-b258-135ebe1e519e-maas-api-tls\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:22.603543 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:22.603508 4493 generic.go:356] "Generic (PLEG): container finished" podID="bc04227c-da6e-421c-b258-135ebe1e519e" containerID="5ed251b06cc3759bea07888a96482fb1abc5fc9f0f78995dcb3ee66a4fd5e710" exitCode=0 Nov 03 21:09:22.604029 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:22.603558 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j" Nov 03 21:09:22.604029 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:22.603595 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j" event={"ID":"bc04227c-da6e-421c-b258-135ebe1e519e","Type":"ContainerDied","Data":"5ed251b06cc3759bea07888a96482fb1abc5fc9f0f78995dcb3ee66a4fd5e710"} Nov 03 21:09:22.604029 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:22.603675 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j" event={"ID":"bc04227c-da6e-421c-b258-135ebe1e519e","Type":"ContainerDied","Data":"b8d182eb8f0842b2002ee561620ba1c40aaead09cc0a36524c00bdddcda3d14a"} Nov 03 21:09:22.604029 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:22.603713 4493 scope.go:122] "RemoveContainer" containerID="5ed251b06cc3759bea07888a96482fb1abc5fc9f0f78995dcb3ee66a4fd5e710" Nov 03 21:09:22.614716 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:22.614695 4493 scope.go:122] "RemoveContainer" containerID="5ed251b06cc3759bea07888a96482fb1abc5fc9f0f78995dcb3ee66a4fd5e710" Nov 03 21:09:22.615078 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:09:22.615056 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ed251b06cc3759bea07888a96482fb1abc5fc9f0f78995dcb3ee66a4fd5e710\": container with ID starting with 5ed251b06cc3759bea07888a96482fb1abc5fc9f0f78995dcb3ee66a4fd5e710 not found: ID does not exist" containerID="5ed251b06cc3759bea07888a96482fb1abc5fc9f0f78995dcb3ee66a4fd5e710" Nov 03 21:09:22.615150 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:22.615089 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ed251b06cc3759bea07888a96482fb1abc5fc9f0f78995dcb3ee66a4fd5e710"} err="failed to get container status \"5ed251b06cc3759bea07888a96482fb1abc5fc9f0f78995dcb3ee66a4fd5e710\": rpc error: code = NotFound desc = could not find container \"5ed251b06cc3759bea07888a96482fb1abc5fc9f0f78995dcb3ee66a4fd5e710\": container with ID starting with 5ed251b06cc3759bea07888a96482fb1abc5fc9f0f78995dcb3ee66a4fd5e710 not found: ID does not exist" Nov 03 21:09:22.774494 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:22.774456 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j"] Nov 03 21:09:22.825843 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:22.825803 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-89k9j"] Nov 03 21:09:24.562305 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:24.562271 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc04227c-da6e-421c-b258-135ebe1e519e" path="/var/lib/kubelet/pods/bc04227c-da6e-421c-b258-135ebe1e519e/volumes" Nov 03 21:09:25.519650 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:25.519600 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" containerID="cri-o://c7564cc61b0556ffe55481e9d2d5b45e68edfd2a509b20d80876ccf8a16885fb" gracePeriod=45 Nov 03 21:09:25.789837 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:25.789810 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 21:09:25.861420 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:25.861389 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/590f00b6-8a20-45ff-8a02-b7e456071057-kube-api-access-rpd5w\" (UniqueName: \"kubernetes.io/projected/590f00b6-8a20-45ff-8a02-b7e456071057-kube-api-access-rpd5w\") pod \"590f00b6-8a20-45ff-8a02-b7e456071057\" (UID: \"590f00b6-8a20-45ff-8a02-b7e456071057\") " Nov 03 21:09:25.861553 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:25.861438 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-model-cache\") pod \"590f00b6-8a20-45ff-8a02-b7e456071057\" (UID: \"590f00b6-8a20-45ff-8a02-b7e456071057\") " Nov 03 21:09:25.861553 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:25.861544 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-dshm\" (UniqueName: \"kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-dshm\") pod \"590f00b6-8a20-45ff-8a02-b7e456071057\" (UID: \"590f00b6-8a20-45ff-8a02-b7e456071057\") " Nov 03 21:09:25.861638 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:25.861579 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-tmp-dir\") pod \"590f00b6-8a20-45ff-8a02-b7e456071057\" (UID: \"590f00b6-8a20-45ff-8a02-b7e456071057\") " Nov 03 21:09:25.861674 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:25.861638 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-home\" (UniqueName: \"kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-home\") pod \"590f00b6-8a20-45ff-8a02-b7e456071057\" (UID: \"590f00b6-8a20-45ff-8a02-b7e456071057\") " Nov 03 21:09:25.861725 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:25.861690 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/590f00b6-8a20-45ff-8a02-b7e456071057-tls-certs\" (UniqueName: \"kubernetes.io/secret/590f00b6-8a20-45ff-8a02-b7e456071057-tls-certs\") pod \"590f00b6-8a20-45ff-8a02-b7e456071057\" (UID: \"590f00b6-8a20-45ff-8a02-b7e456071057\") " Nov 03 21:09:25.861760 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:25.861720 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-model-cache" pod "590f00b6-8a20-45ff-8a02-b7e456071057" (UID: "590f00b6-8a20-45ff-8a02-b7e456071057"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:25.861937 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:25.861912 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-home" pod "590f00b6-8a20-45ff-8a02-b7e456071057" (UID: "590f00b6-8a20-45ff-8a02-b7e456071057"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:25.862083 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:25.861943 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-tmp-dir" pod "590f00b6-8a20-45ff-8a02-b7e456071057" (UID: "590f00b6-8a20-45ff-8a02-b7e456071057"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:25.862314 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:25.862293 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-tmp-dir\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:25.862425 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:25.862318 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-home\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:25.862425 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:25.862349 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-model-cache\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:25.864059 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:25.864038 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590f00b6-8a20-45ff-8a02-b7e456071057-kube-api-access-rpd5w" pod "590f00b6-8a20-45ff-8a02-b7e456071057" (UID: "590f00b6-8a20-45ff-8a02-b7e456071057"). InnerVolumeSpecName "kube-api-access-rpd5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:25.864059 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:25.864041 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/590f00b6-8a20-45ff-8a02-b7e456071057-tls-certs" pod "590f00b6-8a20-45ff-8a02-b7e456071057" (UID: "590f00b6-8a20-45ff-8a02-b7e456071057"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:25.864173 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:25.864084 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-dshm" pod "590f00b6-8a20-45ff-8a02-b7e456071057" (UID: "590f00b6-8a20-45ff-8a02-b7e456071057"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:25.963409 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:25.963371 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590f00b6-8a20-45ff-8a02-b7e456071057-tls-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:25.963409 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:25.963411 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpd5w\" (UniqueName: \"kubernetes.io/projected/590f00b6-8a20-45ff-8a02-b7e456071057-kube-api-access-rpd5w\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:25.963679 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:25.963427 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/590f00b6-8a20-45ff-8a02-b7e456071057-dshm\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:26.631192 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:26.631160 4493 generic.go:356] "Generic (PLEG): container finished" podID="590f00b6-8a20-45ff-8a02-b7e456071057" containerID="c7564cc61b0556ffe55481e9d2d5b45e68edfd2a509b20d80876ccf8a16885fb" exitCode=0 Nov 03 21:09:26.631379 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:26.631217 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" event={"ID":"590f00b6-8a20-45ff-8a02-b7e456071057","Type":"ContainerDied","Data":"c7564cc61b0556ffe55481e9d2d5b45e68edfd2a509b20d80876ccf8a16885fb"} Nov 03 21:09:26.631379 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:26.631243 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" event={"ID":"590f00b6-8a20-45ff-8a02-b7e456071057","Type":"ContainerDied","Data":"ff5ec9dcc2c781a1a8b66bbaeef4a3298e1c5c8f9233cfb30a03146464ff282d"} Nov 03 21:09:26.631379 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:26.631258 4493 scope.go:122] "RemoveContainer" containerID="c7564cc61b0556ffe55481e9d2d5b45e68edfd2a509b20d80876ccf8a16885fb" Nov 03 21:09:26.631379 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:26.631217 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d" Nov 03 21:09:26.642564 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:26.642534 4493 scope.go:122] "RemoveContainer" containerID="c7564cc61b0556ffe55481e9d2d5b45e68edfd2a509b20d80876ccf8a16885fb" Nov 03 21:09:26.642816 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:09:26.642798 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7564cc61b0556ffe55481e9d2d5b45e68edfd2a509b20d80876ccf8a16885fb\": container with ID starting with c7564cc61b0556ffe55481e9d2d5b45e68edfd2a509b20d80876ccf8a16885fb not found: ID does not exist" containerID="c7564cc61b0556ffe55481e9d2d5b45e68edfd2a509b20d80876ccf8a16885fb" Nov 03 21:09:26.642867 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:26.642827 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7564cc61b0556ffe55481e9d2d5b45e68edfd2a509b20d80876ccf8a16885fb"} err="failed to get container status \"c7564cc61b0556ffe55481e9d2d5b45e68edfd2a509b20d80876ccf8a16885fb\": rpc error: code = NotFound desc = could not find container \"c7564cc61b0556ffe55481e9d2d5b45e68edfd2a509b20d80876ccf8a16885fb\": container with ID starting with c7564cc61b0556ffe55481e9d2d5b45e68edfd2a509b20d80876ccf8a16885fb not found: ID does not exist" Nov 03 21:09:26.800872 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:26.800839 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d"] Nov 03 21:09:26.852238 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:26.852155 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-1ec892/distinct-w5-1ec892-kserve-6496b67d7d-27f2d"] Nov 03 21:09:28.093682 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.093650 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7"] Nov 03 21:09:28.094191 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.094178 4493 cpu_manager.go:414] "containerMap: removing container" podUID="8c88cfbc-20ad-47cb-8138-cabbaf57eb1d" containerName="istio-proxy" Nov 03 21:09:28.094254 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.094192 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c88cfbc-20ad-47cb-8138-cabbaf57eb1d" containerName="istio-proxy" Nov 03 21:09:28.094254 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.094215 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" Nov 03 21:09:28.094254 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.094221 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" Nov 03 21:09:28.094254 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.094229 4493 cpu_manager.go:414] "containerMap: removing container" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" Nov 03 21:09:28.094254 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.094234 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" Nov 03 21:09:28.094254 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.094253 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd" containerName="payload-processing" Nov 03 21:09:28.094462 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.094258 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd" containerName="payload-processing" Nov 03 21:09:28.094462 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.094266 4493 cpu_manager.go:414] "containerMap: removing container" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" Nov 03 21:09:28.094462 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.094271 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" Nov 03 21:09:28.094462 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.094279 4493 cpu_manager.go:414] "containerMap: removing container" podUID="cb648add-5f46-4562-83f2-eb0703c98d55" containerName="payload-pre-processing" Nov 03 21:09:28.094462 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.094283 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb648add-5f46-4562-83f2-eb0703c98d55" containerName="payload-pre-processing" Nov 03 21:09:28.094462 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.094290 4493 cpu_manager.go:414] "containerMap: removing container" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" Nov 03 21:09:28.094462 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.094294 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" Nov 03 21:09:28.094462 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.094306 4493 cpu_manager.go:414] "containerMap: removing container" podUID="bc04227c-da6e-421c-b258-135ebe1e519e" containerName="maas-api" Nov 03 21:09:28.094462 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.094310 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc04227c-da6e-421c-b258-135ebe1e519e" containerName="maas-api" Nov 03 21:09:28.094462 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.094428 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c88cfbc-20ad-47cb-8138-cabbaf57eb1d" containerName="istio-proxy" Nov 03 21:09:28.094462 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.094442 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8205aa0-9b8a-4961-bd95-947ac5edfdff" containerName="main" Nov 03 21:09:28.094462 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.094450 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="827fd825-7e75-4174-b522-2de39a8cce28" containerName="main" Nov 03 21:09:28.094462 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.094458 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea2b8063-56a5-4adf-a1eb-a5b7e1b301cd" containerName="payload-processing" Nov 03 21:09:28.094462 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.094464 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" containerName="main" Nov 03 21:09:28.094857 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.094470 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb648add-5f46-4562-83f2-eb0703c98d55" containerName="payload-pre-processing" Nov 03 21:09:28.094857 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.094476 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="46e22045-16c7-48e6-9046-87433500fe27" containerName="main" Nov 03 21:09:28.094857 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.094481 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc04227c-da6e-421c-b258-135ebe1e519e" containerName="maas-api" Nov 03 21:09:28.099150 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.099131 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7" Nov 03 21:09:28.153117 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.153086 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7"] Nov 03 21:09:28.189953 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.189924 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d24cf1-1727-448d-bbad-94d523281bf2-metrics-certs\") pod \"maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7\" (UID: \"b8d24cf1-1727-448d-bbad-94d523281bf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7" Nov 03 21:09:28.190100 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.190049 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d24cf1-1727-448d-bbad-94d523281bf2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7\" (UID: \"b8d24cf1-1727-448d-bbad-94d523281bf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7" Nov 03 21:09:28.190100 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.190093 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8d24cf1-1727-448d-bbad-94d523281bf2-maas-api-tls\") pod \"maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7\" (UID: \"b8d24cf1-1727-448d-bbad-94d523281bf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7" Nov 03 21:09:28.190248 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.190231 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsjkl\" (UniqueName: \"kubernetes.io/projected/b8d24cf1-1727-448d-bbad-94d523281bf2-kube-api-access-tsjkl\") pod \"maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7\" (UID: \"b8d24cf1-1727-448d-bbad-94d523281bf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7" Nov 03 21:09:28.199973 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.199951 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-562eb2-serving-cert\"" Nov 03 21:09:28.291559 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.291520 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d24cf1-1727-448d-bbad-94d523281bf2-metrics-certs\") pod \"maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7\" (UID: \"b8d24cf1-1727-448d-bbad-94d523281bf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7" Nov 03 21:09:28.291831 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.291652 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d24cf1-1727-448d-bbad-94d523281bf2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7\" (UID: \"b8d24cf1-1727-448d-bbad-94d523281bf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7" Nov 03 21:09:28.291831 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.291703 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8d24cf1-1727-448d-bbad-94d523281bf2-maas-api-tls\") pod \"maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7\" (UID: \"b8d24cf1-1727-448d-bbad-94d523281bf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7" Nov 03 21:09:28.291831 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.291808 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsjkl\" (UniqueName: \"kubernetes.io/projected/b8d24cf1-1727-448d-bbad-94d523281bf2-kube-api-access-tsjkl\") pod \"maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7\" (UID: \"b8d24cf1-1727-448d-bbad-94d523281bf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7" Nov 03 21:09:28.291998 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:09:28.291881 4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-serving-cert: secret "maas-api-e2e-inf-b-562eb2-serving-cert" not found Nov 03 21:09:28.291998 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:09:28.291969 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8d24cf1-1727-448d-bbad-94d523281bf2-maas-api-tls podName:b8d24cf1-1727-448d-bbad-94d523281bf2 nodeName:}" failed. No retries permitted until 2026-11-03 21:09:28.791949918 +0000 UTC m=+2440.349940395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8d24cf1-1727-448d-bbad-94d523281bf2-maas-api-tls") pod "maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7" (UID: "b8d24cf1-1727-448d-bbad-94d523281bf2") : secret "maas-api-e2e-inf-b-562eb2-serving-cert" not found Nov 03 21:09:28.292404 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.292387 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d24cf1-1727-448d-bbad-94d523281bf2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7\" (UID: \"b8d24cf1-1727-448d-bbad-94d523281bf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7" Nov 03 21:09:28.294322 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.294302 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d24cf1-1727-448d-bbad-94d523281bf2-metrics-certs\") pod \"maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7\" (UID: \"b8d24cf1-1727-448d-bbad-94d523281bf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7" Nov 03 21:09:28.354237 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.354155 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsjkl\" (UniqueName: \"kubernetes.io/projected/b8d24cf1-1727-448d-bbad-94d523281bf2-kube-api-access-tsjkl\") pod \"maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7\" (UID: \"b8d24cf1-1727-448d-bbad-94d523281bf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7" Nov 03 21:09:28.567080 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.567037 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="590f00b6-8a20-45ff-8a02-b7e456071057" path="/var/lib/kubelet/pods/590f00b6-8a20-45ff-8a02-b7e456071057/volumes" Nov 03 21:09:28.801243 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.801207 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8d24cf1-1727-448d-bbad-94d523281bf2-maas-api-tls\") pod \"maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7\" (UID: \"b8d24cf1-1727-448d-bbad-94d523281bf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7" Nov 03 21:09:28.803754 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:28.803726 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8d24cf1-1727-448d-bbad-94d523281bf2-maas-api-tls\") pod \"maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7\" (UID: \"b8d24cf1-1727-448d-bbad-94d523281bf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7" Nov 03 21:09:29.012717 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:29.012682 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7" Nov 03 21:09:29.393408 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:29.393316 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7"] Nov 03 21:09:29.435605 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:09:29.435571 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d24cf1_1727_448d_bbad_94d523281bf2.slice/crio-2a8e9578961b10436f8a24ce7759294d1ab284e56a222251a2375b39c2a943fe WatchSource:0}: Error finding container 2a8e9578961b10436f8a24ce7759294d1ab284e56a222251a2375b39c2a943fe: Status 404 returned error can't find the container with id 2a8e9578961b10436f8a24ce7759294d1ab284e56a222251a2375b39c2a943fe Nov 03 21:09:29.653933 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:29.653850 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7" event={"ID":"b8d24cf1-1727-448d-bbad-94d523281bf2","Type":"ContainerStarted","Data":"2a8e9578961b10436f8a24ce7759294d1ab284e56a222251a2375b39c2a943fe"} Nov 03 21:09:30.662244 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:30.662167 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7" event={"ID":"b8d24cf1-1727-448d-bbad-94d523281bf2","Type":"ContainerStarted","Data":"1fdd43a8a1a31de1bb19da69a04a4de097228c5794785b7afd8ef846933a8bcf"} Nov 03 21:09:30.662665 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:30.662294 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7" Nov 03 21:09:30.781755 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:30.781690 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7" podStartSLOduration=1.890150229 podStartE2EDuration="2.78167058s" totalImagesPullingTime="891.520351ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:30.778922377 +0000 UTC m=+2442.336912876" watchObservedRunningTime="2026-11-03 21:09:30.78167058 +0000 UTC m=+2442.339661080" Nov 03 21:09:36.675089 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:36.675059 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7" Nov 03 21:09:39.285658 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:39.285626 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r"] Nov 03 21:09:39.286525 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:39.286507 4493 cpu_manager.go:414] "containerMap: removing container" podUID="8c88cfbc-20ad-47cb-8138-cabbaf57eb1d" containerName="istio-proxy" Nov 03 21:09:39.286602 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:39.286526 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c88cfbc-20ad-47cb-8138-cabbaf57eb1d" containerName="istio-proxy" Nov 03 21:09:39.286636 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:39.286611 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c88cfbc-20ad-47cb-8138-cabbaf57eb1d" containerName="istio-proxy" Nov 03 21:09:39.290487 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:39.290464 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r" Nov 03 21:09:39.346903 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:39.346865 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r"] Nov 03 21:09:39.433945 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:39.399566 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc018b8b-3821-4f83-b21a-9480a34aece0-metrics-certs\") pod \"maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r\" (UID: \"bc018b8b-3821-4f83-b21a-9480a34aece0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r" Nov 03 21:09:39.433945 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:39.423522 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-98d1d2-serving-cert\"" Nov 03 21:09:39.433945 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:39.423544 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc018b8b-3821-4f83-b21a-9480a34aece0-maas-api-tls\") pod \"maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r\" (UID: \"bc018b8b-3821-4f83-b21a-9480a34aece0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r" Nov 03 21:09:39.433945 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:39.423638 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc018b8b-3821-4f83-b21a-9480a34aece0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r\" (UID: \"bc018b8b-3821-4f83-b21a-9480a34aece0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r" Nov 03 21:09:39.433945 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:39.424000 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5jct\" (UniqueName: \"kubernetes.io/projected/bc018b8b-3821-4f83-b21a-9480a34aece0-kube-api-access-k5jct\") pod \"maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r\" (UID: \"bc018b8b-3821-4f83-b21a-9480a34aece0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r" Nov 03 21:09:39.531100 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:39.531061 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc018b8b-3821-4f83-b21a-9480a34aece0-metrics-certs\") pod \"maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r\" (UID: \"bc018b8b-3821-4f83-b21a-9480a34aece0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r" Nov 03 21:09:39.531320 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:39.531118 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc018b8b-3821-4f83-b21a-9480a34aece0-maas-api-tls\") pod \"maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r\" (UID: \"bc018b8b-3821-4f83-b21a-9480a34aece0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r" Nov 03 21:09:39.531320 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:39.531164 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc018b8b-3821-4f83-b21a-9480a34aece0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r\" (UID: \"bc018b8b-3821-4f83-b21a-9480a34aece0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r" Nov 03 21:09:39.531500 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:39.531369 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5jct\" (UniqueName: \"kubernetes.io/projected/bc018b8b-3821-4f83-b21a-9480a34aece0-kube-api-access-k5jct\") pod \"maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r\" (UID: \"bc018b8b-3821-4f83-b21a-9480a34aece0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r" Nov 03 21:09:39.531792 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:09:39.531767 4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-serving-cert: secret "maas-api-e2e-inf-b-98d1d2-serving-cert" not found Nov 03 21:09:39.531894 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:09:39.531841 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc018b8b-3821-4f83-b21a-9480a34aece0-maas-api-tls podName:bc018b8b-3821-4f83-b21a-9480a34aece0 nodeName:}" failed. No retries permitted until 2026-11-03 21:09:40.031820033 +0000 UTC m=+2451.589810527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bc018b8b-3821-4f83-b21a-9480a34aece0-maas-api-tls") pod "maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r" (UID: "bc018b8b-3821-4f83-b21a-9480a34aece0") : secret "maas-api-e2e-inf-b-98d1d2-serving-cert" not found Nov 03 21:09:39.533000 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:39.532563 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc018b8b-3821-4f83-b21a-9480a34aece0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r\" (UID: \"bc018b8b-3821-4f83-b21a-9480a34aece0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r" Nov 03 21:09:39.539252 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:39.539190 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc018b8b-3821-4f83-b21a-9480a34aece0-metrics-certs\") pod \"maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r\" (UID: \"bc018b8b-3821-4f83-b21a-9480a34aece0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r" Nov 03 21:09:39.591074 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:39.591038 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5jct\" (UniqueName: \"kubernetes.io/projected/bc018b8b-3821-4f83-b21a-9480a34aece0-kube-api-access-k5jct\") pod \"maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r\" (UID: \"bc018b8b-3821-4f83-b21a-9480a34aece0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r" Nov 03 21:09:40.037081 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:40.037050 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc018b8b-3821-4f83-b21a-9480a34aece0-maas-api-tls\") pod \"maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r\" (UID: \"bc018b8b-3821-4f83-b21a-9480a34aece0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r" Nov 03 21:09:40.039704 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:40.039682 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc018b8b-3821-4f83-b21a-9480a34aece0-maas-api-tls\") pod \"maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r\" (UID: \"bc018b8b-3821-4f83-b21a-9480a34aece0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r" Nov 03 21:09:40.204984 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:40.204954 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r" Nov 03 21:09:40.594669 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:40.592226 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r"] Nov 03 21:09:40.605854 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:09:40.605814 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc018b8b_3821_4f83_b21a_9480a34aece0.slice/crio-2794c9cf1ac54aacc9922d742d418fc18f0811198204f2115d87a48d83db2881 WatchSource:0}: Error finding container 2794c9cf1ac54aacc9922d742d418fc18f0811198204f2115d87a48d83db2881: Status 404 returned error can't find the container with id 2794c9cf1ac54aacc9922d742d418fc18f0811198204f2115d87a48d83db2881 Nov 03 21:09:40.734416 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:40.734376 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r" event={"ID":"bc018b8b-3821-4f83-b21a-9480a34aece0","Type":"ContainerStarted","Data":"2794c9cf1ac54aacc9922d742d418fc18f0811198204f2115d87a48d83db2881"} Nov 03 21:09:41.742955 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:41.742866 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r" event={"ID":"bc018b8b-3821-4f83-b21a-9480a34aece0","Type":"ContainerStarted","Data":"9888a9899a9fef6a92dcde6990ca1d5b22a13870755315ab5d99c00bfe19dca0"} Nov 03 21:09:41.743412 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:41.743073 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r" Nov 03 21:09:41.864021 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:41.863975 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r" podStartSLOduration=1.983195699 podStartE2EDuration="2.863960727s" totalImagesPullingTime="880.765028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:41.862473439 +0000 UTC m=+2453.420463969" watchObservedRunningTime="2026-11-03 21:09:41.863960727 +0000 UTC m=+2453.421951225" Nov 03 21:09:46.224984 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:46.224946 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq"] Nov 03 21:09:46.225481 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:46.225216 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" podUID="21488ac9-bfba-4bb5-ba76-8c76c7c9fa21" containerName="istio-proxy" containerID="cri-o://0ced931349b00e04af2fef095509d00e698fdefb6afeeec03e58a1a2f0cd71a4" gracePeriod=30 Nov 03 21:09:47.756937 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:47.756906 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r" Nov 03 21:09:50.162161 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:50.162135 4493 scope.go:122] "RemoveContainer" containerID="a366ad942960a4fbe767b18240df00480315a431233f8e84c74ad73f79bd837c" Nov 03 21:09:51.502057 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.501997 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:09:51.610090 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.610059 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-workload-certs\") pod \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " Nov 03 21:09:51.610294 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.610124 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-envoy\") pod \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " Nov 03 21:09:51.610294 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.610186 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istiod-ca-cert\") pod \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " Nov 03 21:09:51.610294 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.610238 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-credential-socket\") pod \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " Nov 03 21:09:51.610559 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.610370 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-workload-socket\") pod \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " Nov 03 21:09:51.610559 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.610426 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-data\") pod \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " Nov 03 21:09:51.610559 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.610440 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-workload-certs" pod "21488ac9-bfba-4bb5-ba76-8c76c7c9fa21" (UID: "21488ac9-bfba-4bb5-ba76-8c76c7c9fa21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:51.610559 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.610467 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-token\" (UniqueName: \"kubernetes.io/projected/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-token\") pod \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " Nov 03 21:09:51.610559 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.610545 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-kube-api-access-ftcg6\" (UniqueName: \"kubernetes.io/projected/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-kube-api-access-ftcg6\") pod \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " Nov 03 21:09:51.610892 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.610626 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-podinfo\") pod \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\" (UID: \"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21\") " Nov 03 21:09:51.610892 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.610734 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-credential-socket" pod "21488ac9-bfba-4bb5-ba76-8c76c7c9fa21" (UID: "21488ac9-bfba-4bb5-ba76-8c76c7c9fa21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:51.611010 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.610924 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-data" pod "21488ac9-bfba-4bb5-ba76-8c76c7c9fa21" (UID: "21488ac9-bfba-4bb5-ba76-8c76c7c9fa21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:51.611010 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.610934 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istiod-ca-cert" pod "21488ac9-bfba-4bb5-ba76-8c76c7c9fa21" (UID: "21488ac9-bfba-4bb5-ba76-8c76c7c9fa21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:51.611147 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.611111 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-workload-socket" pod "21488ac9-bfba-4bb5-ba76-8c76c7c9fa21" (UID: "21488ac9-bfba-4bb5-ba76-8c76c7c9fa21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:51.611411 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.611385 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:51.611555 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.611415 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:51.611555 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.611436 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:51.611555 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.611463 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:51.611555 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.611477 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:51.612783 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.612755 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-envoy" pod "21488ac9-bfba-4bb5-ba76-8c76c7c9fa21" (UID: "21488ac9-bfba-4bb5-ba76-8c76c7c9fa21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:51.613000 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.612984 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-kube-api-access-ftcg6" pod "21488ac9-bfba-4bb5-ba76-8c76c7c9fa21" (UID: "21488ac9-bfba-4bb5-ba76-8c76c7c9fa21"). InnerVolumeSpecName "kube-api-access-ftcg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:51.613143 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.613124 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-token" pod "21488ac9-bfba-4bb5-ba76-8c76c7c9fa21" (UID: "21488ac9-bfba-4bb5-ba76-8c76c7c9fa21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:51.613259 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.613246 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-podinfo" pod "21488ac9-bfba-4bb5-ba76-8c76c7c9fa21" (UID: "21488ac9-bfba-4bb5-ba76-8c76c7c9fa21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:09:51.713536 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.713448 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:51.713536 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.713482 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftcg6\" (UniqueName: \"kubernetes.io/projected/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-kube-api-access-ftcg6\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:51.713536 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.713498 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:51.713536 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.713511 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:51.734130 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.734098 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67"] Nov 03 21:09:51.747948 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:09:51.747904 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:09:51.819615 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.819576 4493 generic.go:356] "Generic (PLEG): container finished" podID="21488ac9-bfba-4bb5-ba76-8c76c7c9fa21" containerID="0ced931349b00e04af2fef095509d00e698fdefb6afeeec03e58a1a2f0cd71a4" exitCode=0 Nov 03 21:09:51.819827 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.819625 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" Nov 03 21:09:51.819827 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.819668 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" event={"ID":"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21","Type":"ContainerDied","Data":"0ced931349b00e04af2fef095509d00e698fdefb6afeeec03e58a1a2f0cd71a4"} Nov 03 21:09:51.819827 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.819734 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq" event={"ID":"21488ac9-bfba-4bb5-ba76-8c76c7c9fa21","Type":"ContainerDied","Data":"9ff26beb3da47bfc02d0f087732fc3ca21bdba728917acd5df9b1486814e12ca"} Nov 03 21:09:51.819827 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.819764 4493 scope.go:122] "RemoveContainer" containerID="0ced931349b00e04af2fef095509d00e698fdefb6afeeec03e58a1a2f0cd71a4" Nov 03 21:09:51.832680 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.832451 4493 scope.go:122] "RemoveContainer" containerID="2ebfbf22f684f438ec47b8de4b01b53a8c35ff0ebe31c09ab257425d29f92056" Nov 03 21:09:51.842793 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.842773 4493 scope.go:122] "RemoveContainer" containerID="0ced931349b00e04af2fef095509d00e698fdefb6afeeec03e58a1a2f0cd71a4" Nov 03 21:09:51.843080 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:09:51.843061 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ced931349b00e04af2fef095509d00e698fdefb6afeeec03e58a1a2f0cd71a4\": container with ID starting with 0ced931349b00e04af2fef095509d00e698fdefb6afeeec03e58a1a2f0cd71a4 not found: ID does not exist" containerID="0ced931349b00e04af2fef095509d00e698fdefb6afeeec03e58a1a2f0cd71a4" Nov 03 21:09:51.843135 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.843089 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ced931349b00e04af2fef095509d00e698fdefb6afeeec03e58a1a2f0cd71a4"} err="failed to get container status \"0ced931349b00e04af2fef095509d00e698fdefb6afeeec03e58a1a2f0cd71a4\": rpc error: code = NotFound desc = could not find container \"0ced931349b00e04af2fef095509d00e698fdefb6afeeec03e58a1a2f0cd71a4\": container with ID starting with 0ced931349b00e04af2fef095509d00e698fdefb6afeeec03e58a1a2f0cd71a4 not found: ID does not exist" Nov 03 21:09:51.843135 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.843107 4493 scope.go:122] "RemoveContainer" containerID="2ebfbf22f684f438ec47b8de4b01b53a8c35ff0ebe31c09ab257425d29f92056" Nov 03 21:09:51.843376 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:09:51.843358 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ebfbf22f684f438ec47b8de4b01b53a8c35ff0ebe31c09ab257425d29f92056\": container with ID starting with 2ebfbf22f684f438ec47b8de4b01b53a8c35ff0ebe31c09ab257425d29f92056 not found: ID does not exist" containerID="2ebfbf22f684f438ec47b8de4b01b53a8c35ff0ebe31c09ab257425d29f92056" Nov 03 21:09:51.843417 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.843384 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ebfbf22f684f438ec47b8de4b01b53a8c35ff0ebe31c09ab257425d29f92056"} err="failed to get container status \"2ebfbf22f684f438ec47b8de4b01b53a8c35ff0ebe31c09ab257425d29f92056\": rpc error: code = NotFound desc = could not find container \"2ebfbf22f684f438ec47b8de4b01b53a8c35ff0ebe31c09ab257425d29f92056\": container with ID starting with 2ebfbf22f684f438ec47b8de4b01b53a8c35ff0ebe31c09ab257425d29f92056 not found: ID does not exist" Nov 03 21:09:51.970802 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.970768 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq"] Nov 03 21:09:51.974035 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:51.974016 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c571ef-openshift-default-75fcd94797-hlrpq"] Nov 03 21:09:52.561945 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:09:52.561912 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21488ac9-bfba-4bb5-ba76-8c76c7c9fa21" path="/var/lib/kubelet/pods/21488ac9-bfba-4bb5-ba76-8c76c7c9fa21/volumes" Nov 03 21:10:03.276844 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:03.276806 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn"] Nov 03 21:10:05.830313 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:05.830252 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f"] Nov 03 21:10:05.830313 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:05.830500 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f" podUID="fec5bf6e-548b-4ace-9533-f2f6837fa1a0" containerName="maas-api" containerID="cri-o://2cfd25ebf60d155b5f69a8165c5fe67348b471ec46f938c1c416ebc8dd02db82" gracePeriod=30 Nov 03 21:10:06.130178 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:06.130154 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f" Nov 03 21:10:06.198677 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:06.198644 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-maas-api-tls\") pod \"fec5bf6e-548b-4ace-9533-f2f6837fa1a0\" (UID: \"fec5bf6e-548b-4ace-9533-f2f6837fa1a0\") " Nov 03 21:10:06.198870 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:06.198849 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-openshift-service-ca\") pod \"fec5bf6e-548b-4ace-9533-f2f6837fa1a0\" (UID: \"fec5bf6e-548b-4ace-9533-f2f6837fa1a0\") " Nov 03 21:10:06.198944 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:06.198920 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-kube-api-access-tfpsm\" (UniqueName: \"kubernetes.io/projected/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-kube-api-access-tfpsm\") pod \"fec5bf6e-548b-4ace-9533-f2f6837fa1a0\" (UID: \"fec5bf6e-548b-4ace-9533-f2f6837fa1a0\") " Nov 03 21:10:06.198997 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:06.198965 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-metrics-certs\") pod \"fec5bf6e-548b-4ace-9533-f2f6837fa1a0\" (UID: \"fec5bf6e-548b-4ace-9533-f2f6837fa1a0\") " Nov 03 21:10:06.199250 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:06.199209 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-openshift-service-ca" pod "fec5bf6e-548b-4ace-9533-f2f6837fa1a0" (UID: "fec5bf6e-548b-4ace-9533-f2f6837fa1a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:10:06.199833 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:06.199805 4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-openshift-service-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:06.201932 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:06.201701 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-kube-api-access-tfpsm" pod "fec5bf6e-548b-4ace-9533-f2f6837fa1a0" (UID: "fec5bf6e-548b-4ace-9533-f2f6837fa1a0"). InnerVolumeSpecName "kube-api-access-tfpsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:06.201932 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:06.201835 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-maas-api-tls" pod "fec5bf6e-548b-4ace-9533-f2f6837fa1a0" (UID: "fec5bf6e-548b-4ace-9533-f2f6837fa1a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:10:06.202393 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:06.202365 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-metrics-certs" pod "fec5bf6e-548b-4ace-9533-f2f6837fa1a0" (UID: "fec5bf6e-548b-4ace-9533-f2f6837fa1a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:10:06.301234 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:06.301199 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfpsm\" (UniqueName: \"kubernetes.io/projected/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-kube-api-access-tfpsm\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:06.301234 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:06.301231 4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-metrics-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:06.301234 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:06.301240 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec5bf6e-548b-4ace-9533-f2f6837fa1a0-maas-api-tls\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:06.757041 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:06.757004 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" containerID="cri-o://767ac478a1db0e744c9a4ad9746eb9e05c880ed5e6f8c0ee595e724fa695e312" gracePeriod=45 Nov 03 21:10:06.761393 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:06.761361 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": EOF" Nov 03 21:10:06.920549 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:06.920515 4493 generic.go:356] "Generic (PLEG): container finished" podID="fec5bf6e-548b-4ace-9533-f2f6837fa1a0" containerID="2cfd25ebf60d155b5f69a8165c5fe67348b471ec46f938c1c416ebc8dd02db82" exitCode=0 Nov 03 21:10:06.920931 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:06.920707 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f" event={"ID":"fec5bf6e-548b-4ace-9533-f2f6837fa1a0","Type":"ContainerDied","Data":"2cfd25ebf60d155b5f69a8165c5fe67348b471ec46f938c1c416ebc8dd02db82"} Nov 03 21:10:06.920931 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:06.920751 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f" event={"ID":"fec5bf6e-548b-4ace-9533-f2f6837fa1a0","Type":"ContainerDied","Data":"c651de485d1a0bf8884b2f8b542187773263839099cb837b820f33c6410f2b22"} Nov 03 21:10:06.920931 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:06.920773 4493 scope.go:122] "RemoveContainer" containerID="2cfd25ebf60d155b5f69a8165c5fe67348b471ec46f938c1c416ebc8dd02db82" Nov 03 21:10:06.920931 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:06.920879 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f" Nov 03 21:10:06.925517 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:06.925486 4493 generic.go:356] "Generic (PLEG): container finished" podID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerID="767ac478a1db0e744c9a4ad9746eb9e05c880ed5e6f8c0ee595e724fa695e312" exitCode=0 Nov 03 21:10:06.925724 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:06.925552 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" event={"ID":"50477b11-be5c-4d14-b7ad-663dd674f82e","Type":"ContainerDied","Data":"767ac478a1db0e744c9a4ad9746eb9e05c880ed5e6f8c0ee595e724fa695e312"} Nov 03 21:10:06.934481 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:06.933887 4493 scope.go:122] "RemoveContainer" containerID="2cfd25ebf60d155b5f69a8165c5fe67348b471ec46f938c1c416ebc8dd02db82" Nov 03 21:10:06.934481 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:10:06.934291 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cfd25ebf60d155b5f69a8165c5fe67348b471ec46f938c1c416ebc8dd02db82\": container with ID starting with 2cfd25ebf60d155b5f69a8165c5fe67348b471ec46f938c1c416ebc8dd02db82 not found: ID does not exist" containerID="2cfd25ebf60d155b5f69a8165c5fe67348b471ec46f938c1c416ebc8dd02db82" Nov 03 21:10:06.934481 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:06.934376 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cfd25ebf60d155b5f69a8165c5fe67348b471ec46f938c1c416ebc8dd02db82"} err="failed to get container status \"2cfd25ebf60d155b5f69a8165c5fe67348b471ec46f938c1c416ebc8dd02db82\": rpc error: code = NotFound desc = could not find container \"2cfd25ebf60d155b5f69a8165c5fe67348b471ec46f938c1c416ebc8dd02db82\": container with ID starting with 2cfd25ebf60d155b5f69a8165c5fe67348b471ec46f938c1c416ebc8dd02db82 not found: ID does not exist" Nov 03 21:10:07.032799 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:07.032776 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 21:10:07.094698 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:07.094658 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f"] Nov 03 21:10:07.112365 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:07.112265 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-home\" (UniqueName: \"kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-home\") pod \"50477b11-be5c-4d14-b7ad-663dd674f82e\" (UID: \"50477b11-be5c-4d14-b7ad-663dd674f82e\") " Nov 03 21:10:07.112365 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:07.112311 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-model-cache\") pod \"50477b11-be5c-4d14-b7ad-663dd674f82e\" (UID: \"50477b11-be5c-4d14-b7ad-663dd674f82e\") " Nov 03 21:10:07.112576 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:07.112395 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-dshm\") pod \"50477b11-be5c-4d14-b7ad-663dd674f82e\" (UID: \"50477b11-be5c-4d14-b7ad-663dd674f82e\") " Nov 03 21:10:07.112576 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:07.112453 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50477b11-be5c-4d14-b7ad-663dd674f82e-kube-api-access-86tll\" (UniqueName: \"kubernetes.io/projected/50477b11-be5c-4d14-b7ad-663dd674f82e-kube-api-access-86tll\") pod \"50477b11-be5c-4d14-b7ad-663dd674f82e\" (UID: \"50477b11-be5c-4d14-b7ad-663dd674f82e\") " Nov 03 21:10:07.112576 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:07.112494 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-tmp-dir\") pod \"50477b11-be5c-4d14-b7ad-663dd674f82e\" (UID: \"50477b11-be5c-4d14-b7ad-663dd674f82e\") " Nov 03 21:10:07.112738 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:07.112592 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50477b11-be5c-4d14-b7ad-663dd674f82e-tls-certs\" (UniqueName: \"kubernetes.io/secret/50477b11-be5c-4d14-b7ad-663dd674f82e-tls-certs\") pod \"50477b11-be5c-4d14-b7ad-663dd674f82e\" (UID: \"50477b11-be5c-4d14-b7ad-663dd674f82e\") " Nov 03 21:10:07.112738 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:07.112671 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-home" pod "50477b11-be5c-4d14-b7ad-663dd674f82e" (UID: "50477b11-be5c-4d14-b7ad-663dd674f82e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:07.112738 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:07.112676 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-model-cache" pod "50477b11-be5c-4d14-b7ad-663dd674f82e" (UID: "50477b11-be5c-4d14-b7ad-663dd674f82e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:07.112864 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:07.112842 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-tmp-dir" pod "50477b11-be5c-4d14-b7ad-663dd674f82e" (UID: "50477b11-be5c-4d14-b7ad-663dd674f82e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:07.113138 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:07.113119 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-tmp-dir\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:07.113438 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:07.113142 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-home\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:07.113438 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:07.113153 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-model-cache\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:07.114827 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:07.114808 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50477b11-be5c-4d14-b7ad-663dd674f82e-tls-certs" pod "50477b11-be5c-4d14-b7ad-663dd674f82e" (UID: "50477b11-be5c-4d14-b7ad-663dd674f82e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:10:07.114921 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:07.114856 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50477b11-be5c-4d14-b7ad-663dd674f82e-kube-api-access-86tll" pod "50477b11-be5c-4d14-b7ad-663dd674f82e" (UID: "50477b11-be5c-4d14-b7ad-663dd674f82e"). InnerVolumeSpecName "kube-api-access-86tll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:07.114921 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:07.114900 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-dshm" pod "50477b11-be5c-4d14-b7ad-663dd674f82e" (UID: "50477b11-be5c-4d14-b7ad-663dd674f82e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:07.146192 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:07.146165 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a19980-74649996d7-54s7f"] Nov 03 21:10:07.218985 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:07.215013 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50477b11-be5c-4d14-b7ad-663dd674f82e-dshm\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:07.218985 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:07.215048 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86tll\" (UniqueName: \"kubernetes.io/projected/50477b11-be5c-4d14-b7ad-663dd674f82e-kube-api-access-86tll\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:07.218985 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:07.215065 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50477b11-be5c-4d14-b7ad-663dd674f82e-tls-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:07.932846 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:07.932820 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" Nov 03 21:10:07.933264 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:07.932847 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67" event={"ID":"50477b11-be5c-4d14-b7ad-663dd674f82e","Type":"ContainerDied","Data":"977f64257f908f57611b5eb14d29d021e946c5c93e811e42ad230240386cbd87"} Nov 03 21:10:07.933264 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:07.932895 4493 scope.go:122] "RemoveContainer" containerID="767ac478a1db0e744c9a4ad9746eb9e05c880ed5e6f8c0ee595e724fa695e312" Nov 03 21:10:08.103919 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:08.103881 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67"] Nov 03 21:10:08.155163 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:08.155132 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-fd9afe/distinct-w0-fd9afe-kserve-56449c6948-hzd67"] Nov 03 21:10:08.570776 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:08.570743 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" path="/var/lib/kubelet/pods/50477b11-be5c-4d14-b7ad-663dd674f82e/volumes" Nov 03 21:10:08.571192 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:08.571176 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fec5bf6e-548b-4ace-9533-f2f6837fa1a0" path="/var/lib/kubelet/pods/fec5bf6e-548b-4ace-9533-f2f6837fa1a0/volumes" Nov 03 21:10:12.638244 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:12.638204 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f"] Nov 03 21:10:12.638835 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:12.638459 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" podUID="49c1d4d0-289d-4028-a3c5-b16bbeb87b40" containerName="istio-proxy" containerID="cri-o://5edae911cd698dc7e0f4ada4ec0f2012a59409673a613caf815a12fbd110df38" gracePeriod=30 Nov 03 21:10:12.653255 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:12.653230 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n"] Nov 03 21:10:12.653482 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:12.653449 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" podUID="701f971e-5fbb-4f4b-81ec-1dfa66005128" containerName="istio-proxy" containerID="cri-o://92cc2bacbc8339474b32c083db15dd64c6d0de787073729b42004110badb9e15" gracePeriod=30 Nov 03 21:10:12.684005 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:12.683978 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv"] Nov 03 21:10:12.684232 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:12.684162 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" podUID="7ec5039b-2ca6-4099-9919-12f4b42af9c7" containerName="istio-proxy" containerID="cri-o://b1bc003adba6e70ee97b1466a631887ae342b6ebad77469d5707efa48ed0139c" gracePeriod=30 Nov 03 21:10:16.663587 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:16.663554 4493 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:10:16.664004 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:16.663616 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" podUID="701f971e-5fbb-4f4b-81ec-1dfa66005128" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:10:18.004608 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.004153 4493 generic.go:356] "Generic (PLEG): container finished" podID="701f971e-5fbb-4f4b-81ec-1dfa66005128" containerID="92cc2bacbc8339474b32c083db15dd64c6d0de787073729b42004110badb9e15" exitCode=0 Nov 03 21:10:18.004608 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.004277 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" event={"ID":"701f971e-5fbb-4f4b-81ec-1dfa66005128","Type":"ContainerDied","Data":"92cc2bacbc8339474b32c083db15dd64c6d0de787073729b42004110badb9e15"} Nov 03 21:10:18.004608 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.004354 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" event={"ID":"701f971e-5fbb-4f4b-81ec-1dfa66005128","Type":"ContainerDied","Data":"e503d470afe4db80a3011ed18ebb0df8ceb336493ddf7d71724edde29a1b98c1"} Nov 03 21:10:18.004608 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.004382 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e503d470afe4db80a3011ed18ebb0df8ceb336493ddf7d71724edde29a1b98c1" Nov 03 21:10:18.004608 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.004402 4493 scope.go:122] "RemoveContainer" containerID="73764b575f50c5e1e9b4d1ff2f3b97c84f749f0cfe430d47e02c385ea31812a8" Nov 03 21:10:18.007821 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.007797 4493 generic.go:356] "Generic (PLEG): container finished" podID="49c1d4d0-289d-4028-a3c5-b16bbeb87b40" containerID="5edae911cd698dc7e0f4ada4ec0f2012a59409673a613caf815a12fbd110df38" exitCode=0 Nov 03 21:10:18.008045 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.007970 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" event={"ID":"49c1d4d0-289d-4028-a3c5-b16bbeb87b40","Type":"ContainerDied","Data":"5edae911cd698dc7e0f4ada4ec0f2012a59409673a613caf815a12fbd110df38"} Nov 03 21:10:18.012973 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.012327 4493 generic.go:356] "Generic (PLEG): container finished" podID="7ec5039b-2ca6-4099-9919-12f4b42af9c7" containerID="b1bc003adba6e70ee97b1466a631887ae342b6ebad77469d5707efa48ed0139c" exitCode=0 Nov 03 21:10:18.012973 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.012570 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" event={"ID":"7ec5039b-2ca6-4099-9919-12f4b42af9c7","Type":"ContainerDied","Data":"b1bc003adba6e70ee97b1466a631887ae342b6ebad77469d5707efa48ed0139c"} Nov 03 21:10:18.018445 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.018297 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 21:10:18.052622 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.052595 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:10:18.074573 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.074551 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:10:18.077218 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.077201 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-podinfo\") pod \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " Nov 03 21:10:18.077281 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.077229 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-credential-socket\") pod \"701f971e-5fbb-4f4b-81ec-1dfa66005128\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " Nov 03 21:10:18.077281 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.077247 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-workload-certs\") pod \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " Nov 03 21:10:18.077375 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.077277 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-data\") pod \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " Nov 03 21:10:18.077375 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.077316 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-credential-socket\") pod \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " Nov 03 21:10:18.077469 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.077452 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-envoy\") pod \"701f971e-5fbb-4f4b-81ec-1dfa66005128\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " Nov 03 21:10:18.077524 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.077498 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-token\" (UniqueName: \"kubernetes.io/projected/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-token\") pod \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " Nov 03 21:10:18.077524 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.077510 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-credential-socket" pod "701f971e-5fbb-4f4b-81ec-1dfa66005128" (UID: "701f971e-5fbb-4f4b-81ec-1dfa66005128"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.077631 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.077529 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-workload-certs" pod "7ec5039b-2ca6-4099-9919-12f4b42af9c7" (UID: "7ec5039b-2ca6-4099-9919-12f4b42af9c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.077631 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.077531 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-workload-socket\") pod \"701f971e-5fbb-4f4b-81ec-1dfa66005128\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " Nov 03 21:10:18.077631 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.077627 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-podinfo\") pod \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " Nov 03 21:10:18.077785 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.077623 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-data" pod "7ec5039b-2ca6-4099-9919-12f4b42af9c7" (UID: "7ec5039b-2ca6-4099-9919-12f4b42af9c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.077785 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.077654 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-credential-socket" pod "7ec5039b-2ca6-4099-9919-12f4b42af9c7" (UID: "7ec5039b-2ca6-4099-9919-12f4b42af9c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.077785 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.077672 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-envoy\") pod \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " Nov 03 21:10:18.077785 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.077695 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-token\" (UniqueName: \"kubernetes.io/projected/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-token\") pod \"701f971e-5fbb-4f4b-81ec-1dfa66005128\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " Nov 03 21:10:18.077785 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.077727 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-credential-socket\") pod \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " Nov 03 21:10:18.077785 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.077756 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-token\" (UniqueName: \"kubernetes.io/projected/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-token\") pod \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " Nov 03 21:10:18.077785 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.077780 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ec5039b-2ca6-4099-9919-12f4b42af9c7-kube-api-access-847jx\" (UniqueName: \"kubernetes.io/projected/7ec5039b-2ca6-4099-9919-12f4b42af9c7-kube-api-access-847jx\") pod \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " Nov 03 21:10:18.078164 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.077799 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-workload-certs\") pod \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " Nov 03 21:10:18.078164 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.077867 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-podinfo\") pod \"701f971e-5fbb-4f4b-81ec-1dfa66005128\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " Nov 03 21:10:18.078164 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.077896 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/701f971e-5fbb-4f4b-81ec-1dfa66005128-kube-api-access-g26wv\" (UniqueName: \"kubernetes.io/projected/701f971e-5fbb-4f4b-81ec-1dfa66005128-kube-api-access-g26wv\") pod \"701f971e-5fbb-4f4b-81ec-1dfa66005128\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " Nov 03 21:10:18.078164 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.077949 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-workload-certs\") pod \"701f971e-5fbb-4f4b-81ec-1dfa66005128\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " Nov 03 21:10:18.078164 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.078015 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-data\") pod \"701f971e-5fbb-4f4b-81ec-1dfa66005128\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " Nov 03 21:10:18.078164 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.078047 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istiod-ca-cert\") pod \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " Nov 03 21:10:18.078164 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.078076 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-credential-socket" pod "49c1d4d0-289d-4028-a3c5-b16bbeb87b40" (UID: "49c1d4d0-289d-4028-a3c5-b16bbeb87b40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.078164 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.078112 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-kube-api-access-xs4lv\" (UniqueName: \"kubernetes.io/projected/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-kube-api-access-xs4lv\") pod \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " Nov 03 21:10:18.078164 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.078140 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-envoy\") pod \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " Nov 03 21:10:18.078853 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.078175 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-workload-socket\") pod \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " Nov 03 21:10:18.078853 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.078430 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/701f971e-5fbb-4f4b-81ec-1dfa66005128-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701f971e-5fbb-4f4b-81ec-1dfa66005128-istiod-ca-cert\") pod \"701f971e-5fbb-4f4b-81ec-1dfa66005128\" (UID: \"701f971e-5fbb-4f4b-81ec-1dfa66005128\") " Nov 03 21:10:18.078853 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.078470 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-data\") pod \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " Nov 03 21:10:18.078853 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.078508 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istiod-ca-cert\") pod \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\" (UID: \"49c1d4d0-289d-4028-a3c5-b16bbeb87b40\") " Nov 03 21:10:18.078853 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.078589 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-workload-socket\") pod \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\" (UID: \"7ec5039b-2ca6-4099-9919-12f4b42af9c7\") " Nov 03 21:10:18.079115 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.078917 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-workload-certs" pod "49c1d4d0-289d-4028-a3c5-b16bbeb87b40" (UID: "49c1d4d0-289d-4028-a3c5-b16bbeb87b40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.079288 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.079263 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.079398 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.079295 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.079398 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.079311 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.079398 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.079326 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.079398 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.079359 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.079398 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.079372 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.079669 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.079532 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/701f971e-5fbb-4f4b-81ec-1dfa66005128-istiod-ca-cert" pod "701f971e-5fbb-4f4b-81ec-1dfa66005128" (UID: "701f971e-5fbb-4f4b-81ec-1dfa66005128"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:10:18.079669 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.079603 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-workload-socket" pod "49c1d4d0-289d-4028-a3c5-b16bbeb87b40" (UID: "49c1d4d0-289d-4028-a3c5-b16bbeb87b40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.080373 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.079835 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istiod-ca-cert" pod "49c1d4d0-289d-4028-a3c5-b16bbeb87b40" (UID: "49c1d4d0-289d-4028-a3c5-b16bbeb87b40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:10:18.080373 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.079991 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-data" pod "701f971e-5fbb-4f4b-81ec-1dfa66005128" (UID: "701f971e-5fbb-4f4b-81ec-1dfa66005128"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.080373 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.080049 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-data" pod "49c1d4d0-289d-4028-a3c5-b16bbeb87b40" (UID: "49c1d4d0-289d-4028-a3c5-b16bbeb87b40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.080373 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.080253 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-workload-certs" pod "701f971e-5fbb-4f4b-81ec-1dfa66005128" (UID: "701f971e-5fbb-4f4b-81ec-1dfa66005128"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.080373 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.080312 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istiod-ca-cert" pod "7ec5039b-2ca6-4099-9919-12f4b42af9c7" (UID: "7ec5039b-2ca6-4099-9919-12f4b42af9c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:10:18.080809 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.080472 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-workload-socket" pod "701f971e-5fbb-4f4b-81ec-1dfa66005128" (UID: "701f971e-5fbb-4f4b-81ec-1dfa66005128"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.080809 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.080523 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-workload-socket" pod "7ec5039b-2ca6-4099-9919-12f4b42af9c7" (UID: "7ec5039b-2ca6-4099-9919-12f4b42af9c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.080809 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.080586 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-podinfo" pod "7ec5039b-2ca6-4099-9919-12f4b42af9c7" (UID: "7ec5039b-2ca6-4099-9919-12f4b42af9c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:10:18.080809 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.080633 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-envoy" pod "701f971e-5fbb-4f4b-81ec-1dfa66005128" (UID: "701f971e-5fbb-4f4b-81ec-1dfa66005128"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.080809 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.080645 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-podinfo" pod "49c1d4d0-289d-4028-a3c5-b16bbeb87b40" (UID: "49c1d4d0-289d-4028-a3c5-b16bbeb87b40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:10:18.081264 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.081015 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-token" pod "49c1d4d0-289d-4028-a3c5-b16bbeb87b40" (UID: "49c1d4d0-289d-4028-a3c5-b16bbeb87b40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:18.081814 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.081781 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-envoy" pod "7ec5039b-2ca6-4099-9919-12f4b42af9c7" (UID: "7ec5039b-2ca6-4099-9919-12f4b42af9c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.082289 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.082264 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-token" pod "701f971e-5fbb-4f4b-81ec-1dfa66005128" (UID: "701f971e-5fbb-4f4b-81ec-1dfa66005128"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:18.082722 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.082698 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-podinfo" pod "701f971e-5fbb-4f4b-81ec-1dfa66005128" (UID: "701f971e-5fbb-4f4b-81ec-1dfa66005128"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:10:18.082813 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.082747 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-token" pod "7ec5039b-2ca6-4099-9919-12f4b42af9c7" (UID: "7ec5039b-2ca6-4099-9919-12f4b42af9c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:18.082990 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.082972 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-kube-api-access-xs4lv" pod "49c1d4d0-289d-4028-a3c5-b16bbeb87b40" (UID: "49c1d4d0-289d-4028-a3c5-b16bbeb87b40"). InnerVolumeSpecName "kube-api-access-xs4lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:18.083203 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.083189 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec5039b-2ca6-4099-9919-12f4b42af9c7-kube-api-access-847jx" pod "7ec5039b-2ca6-4099-9919-12f4b42af9c7" (UID: "7ec5039b-2ca6-4099-9919-12f4b42af9c7"). InnerVolumeSpecName "kube-api-access-847jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:18.083275 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.083262 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-envoy" pod "49c1d4d0-289d-4028-a3c5-b16bbeb87b40" (UID: "49c1d4d0-289d-4028-a3c5-b16bbeb87b40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.083926 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.083902 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/701f971e-5fbb-4f4b-81ec-1dfa66005128-kube-api-access-g26wv" pod "701f971e-5fbb-4f4b-81ec-1dfa66005128" (UID: "701f971e-5fbb-4f4b-81ec-1dfa66005128"). InnerVolumeSpecName "kube-api-access-g26wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:18.180171 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.180139 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.180171 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.180166 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.180171 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.180177 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.180447 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.180186 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.180447 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.180195 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.180447 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.180205 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.180447 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.180214 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-847jx\" (UniqueName: \"kubernetes.io/projected/7ec5039b-2ca6-4099-9919-12f4b42af9c7-kube-api-access-847jx\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.180447 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.180225 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.180447 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.180233 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g26wv\" (UniqueName: \"kubernetes.io/projected/701f971e-5fbb-4f4b-81ec-1dfa66005128-kube-api-access-g26wv\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.180447 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.180242 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.180447 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.180250 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.180447 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.180259 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.180447 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.180268 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs4lv\" (UniqueName: \"kubernetes.io/projected/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-kube-api-access-xs4lv\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.180447 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.180277 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.180447 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.180286 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.180447 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.180294 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701f971e-5fbb-4f4b-81ec-1dfa66005128-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.180447 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.180303 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.180447 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.180311 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c1d4d0-289d-4028-a3c5-b16bbeb87b40-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.180447 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.180320 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec5039b-2ca6-4099-9919-12f4b42af9c7-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.180447 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.180328 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec5039b-2ca6-4099-9919-12f4b42af9c7-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.180447 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.180364 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701f971e-5fbb-4f4b-81ec-1dfa66005128-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.298400 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.298314 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" containerID="cri-o://416f6a8fded2e51851deddc103fa9b38c6b31eb80c4e14de164f4a7902da7579" gracePeriod=45 Nov 03 21:10:18.654296 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.654267 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 21:10:18.690530 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.690499 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-tls-certs\" (UniqueName: \"kubernetes.io/secret/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-tls-certs\") pod \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\" (UID: \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\") " Nov 03 21:10:18.690692 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.690544 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-dshm\") pod \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\" (UID: \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\") " Nov 03 21:10:18.690692 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.690576 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-model-cache\") pod \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\" (UID: \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\") " Nov 03 21:10:18.690692 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.690628 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-home\" (UniqueName: \"kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-home\") pod \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\" (UID: \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\") " Nov 03 21:10:18.690867 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.690695 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-tmp-dir\") pod \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\" (UID: \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\") " Nov 03 21:10:18.690930 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.690871 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-kube-api-access-dppvq\" (UniqueName: \"kubernetes.io/projected/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-kube-api-access-dppvq\") pod \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\" (UID: \"d84d7f0a-8ffa-4211-a9be-d73ab3853b40\") " Nov 03 21:10:18.691005 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.690976 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-model-cache" pod "d84d7f0a-8ffa-4211-a9be-d73ab3853b40" (UID: "d84d7f0a-8ffa-4211-a9be-d73ab3853b40"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.691056 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.690991 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-home" pod "d84d7f0a-8ffa-4211-a9be-d73ab3853b40" (UID: "d84d7f0a-8ffa-4211-a9be-d73ab3853b40"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.691056 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.691020 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-tmp-dir" pod "d84d7f0a-8ffa-4211-a9be-d73ab3853b40" (UID: "d84d7f0a-8ffa-4211-a9be-d73ab3853b40"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.691313 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.691293 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-model-cache\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.691611 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.691314 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-home\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.691611 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.691346 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-tmp-dir\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.693030 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.693001 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-dshm" pod "d84d7f0a-8ffa-4211-a9be-d73ab3853b40" (UID: "d84d7f0a-8ffa-4211-a9be-d73ab3853b40"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.693123 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.693029 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-tls-certs" pod "d84d7f0a-8ffa-4211-a9be-d73ab3853b40" (UID: "d84d7f0a-8ffa-4211-a9be-d73ab3853b40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:10:18.693123 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.693083 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-kube-api-access-dppvq" pod "d84d7f0a-8ffa-4211-a9be-d73ab3853b40" (UID: "d84d7f0a-8ffa-4211-a9be-d73ab3853b40"). InnerVolumeSpecName "kube-api-access-dppvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:18.792944 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.792908 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dppvq\" (UniqueName: \"kubernetes.io/projected/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-kube-api-access-dppvq\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.792944 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.792948 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-tls-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.793134 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:18.792962 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d84d7f0a-8ffa-4211-a9be-d73ab3853b40-dshm\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:19.022452 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:19.022360 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" Nov 03 21:10:19.022452 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:19.022377 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f" event={"ID":"49c1d4d0-289d-4028-a3c5-b16bbeb87b40","Type":"ContainerDied","Data":"c32fb4556c3463c4fad8fb4c3e45a3dffba834a29e8eb2748540eaddaacb0cca"} Nov 03 21:10:19.022941 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:19.022453 4493 scope.go:122] "RemoveContainer" containerID="5edae911cd698dc7e0f4ada4ec0f2012a59409673a613caf815a12fbd110df38" Nov 03 21:10:19.024306 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:19.024249 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" Nov 03 21:10:19.024306 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:19.024281 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv" event={"ID":"7ec5039b-2ca6-4099-9919-12f4b42af9c7","Type":"ContainerDied","Data":"78c9f1cef8b2dc9360f038aafd810923d4874af421b7e9b607d47ee67bb30aca"} Nov 03 21:10:19.028983 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:19.028947 4493 generic.go:356] "Generic (PLEG): container finished" podID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerID="416f6a8fded2e51851deddc103fa9b38c6b31eb80c4e14de164f4a7902da7579" exitCode=0 Nov 03 21:10:19.029163 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:19.029143 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" event={"ID":"d84d7f0a-8ffa-4211-a9be-d73ab3853b40","Type":"ContainerDied","Data":"416f6a8fded2e51851deddc103fa9b38c6b31eb80c4e14de164f4a7902da7579"} Nov 03 21:10:19.029224 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:19.029182 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" event={"ID":"d84d7f0a-8ffa-4211-a9be-d73ab3853b40","Type":"ContainerDied","Data":"f2e8dbd23c150069b1b9f5632125a64228ed35ceaf8b33f1dcf43c65efc715ba"} Nov 03 21:10:19.029275 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:19.029243 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" Nov 03 21:10:19.034253 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:19.034230 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n" Nov 03 21:10:19.037309 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:19.037299 4493 scope.go:122] "RemoveContainer" containerID="b1bc003adba6e70ee97b1466a631887ae342b6ebad77469d5707efa48ed0139c" Nov 03 21:10:19.046852 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:19.046832 4493 scope.go:122] "RemoveContainer" containerID="416f6a8fded2e51851deddc103fa9b38c6b31eb80c4e14de164f4a7902da7579" Nov 03 21:10:19.054931 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:19.054911 4493 scope.go:122] "RemoveContainer" containerID="416f6a8fded2e51851deddc103fa9b38c6b31eb80c4e14de164f4a7902da7579" Nov 03 21:10:19.055183 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:10:19.055165 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"416f6a8fded2e51851deddc103fa9b38c6b31eb80c4e14de164f4a7902da7579\": container with ID starting with 416f6a8fded2e51851deddc103fa9b38c6b31eb80c4e14de164f4a7902da7579 not found: ID does not exist" containerID="416f6a8fded2e51851deddc103fa9b38c6b31eb80c4e14de164f4a7902da7579" Nov 03 21:10:19.055229 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:19.055196 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"416f6a8fded2e51851deddc103fa9b38c6b31eb80c4e14de164f4a7902da7579"} err="failed to get container status \"416f6a8fded2e51851deddc103fa9b38c6b31eb80c4e14de164f4a7902da7579\": rpc error: code = NotFound desc = could not find container \"416f6a8fded2e51851deddc103fa9b38c6b31eb80c4e14de164f4a7902da7579\": container with ID starting with 416f6a8fded2e51851deddc103fa9b38c6b31eb80c4e14de164f4a7902da7579 not found: ID does not exist" Nov 03 21:10:19.193990 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:19.193954 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn"] Nov 03 21:10:19.197075 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:19.197053 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn"] Nov 03 21:10:19.313422 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:19.313393 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f"] Nov 03 21:10:19.317720 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:19.317697 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-65r9f"] Nov 03 21:10:19.499052 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:19.499021 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n"] Nov 03 21:10:19.501666 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:19.501641 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lcn7n"] Nov 03 21:10:19.629398 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:19.629290 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv"] Nov 03 21:10:19.634357 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:19.634304 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/premium-w2-b119f2-kserve-567bcf7f49-fthxn" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:10:19.645821 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:19.645785 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m2tfv"] Nov 03 21:10:20.562068 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:20.562039 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49c1d4d0-289d-4028-a3c5-b16bbeb87b40" path="/var/lib/kubelet/pods/49c1d4d0-289d-4028-a3c5-b16bbeb87b40/volumes" Nov 03 21:10:20.562524 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:20.562510 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="701f971e-5fbb-4f4b-81ec-1dfa66005128" path="/var/lib/kubelet/pods/701f971e-5fbb-4f4b-81ec-1dfa66005128/volumes" Nov 03 21:10:20.562981 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:20.562968 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ec5039b-2ca6-4099-9919-12f4b42af9c7" path="/var/lib/kubelet/pods/7ec5039b-2ca6-4099-9919-12f4b42af9c7/volumes" Nov 03 21:10:20.563701 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:20.563686 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" path="/var/lib/kubelet/pods/d84d7f0a-8ffa-4211-a9be-d73ab3853b40/volumes" Nov 03 21:10:21.745075 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:10:21.745031 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:10:50.213113 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:10:50.213074 4493 scope.go:122] "RemoveContainer" containerID="92cc2bacbc8339474b32c083db15dd64c6d0de787073729b42004110badb9e15" Nov 03 21:10:51.745688 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:10:51.745646 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:11:08.437292 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:08.437246 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r"] Nov 03 21:11:08.437903 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:08.437609 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" podUID="f8dec249-9f2a-4bfd-b83b-f72d50e5b936" containerName="istio-proxy" containerID="cri-o://742bad4705c31054cc620bed93507bf6120cc4755ecc17274c4da1d3e5a2e4de" gracePeriod=30 Nov 03 21:11:13.708146 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.708090 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 21:11:13.855272 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.855243 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-token\" (UniqueName: \"kubernetes.io/projected/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-token\") pod \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " Nov 03 21:11:13.855455 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.855299 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-data\") pod \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " Nov 03 21:11:13.855455 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.855351 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-workload-socket\") pod \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " Nov 03 21:11:13.855455 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.855395 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-kube-api-access-z7w6r\" (UniqueName: \"kubernetes.io/projected/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-kube-api-access-z7w6r\") pod \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " Nov 03 21:11:13.855455 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.855426 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-podinfo\") pod \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " Nov 03 21:11:13.855683 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.855457 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-envoy\") pod \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " Nov 03 21:11:13.855683 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.855506 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-credential-socket\") pod \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " Nov 03 21:11:13.855683 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.855604 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istiod-ca-cert\") pod \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " Nov 03 21:11:13.855683 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.855655 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-workload-socket" pod "f8dec249-9f2a-4bfd-b83b-f72d50e5b936" (UID: "f8dec249-9f2a-4bfd-b83b-f72d50e5b936"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:11:13.855683 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.855674 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-data" pod "f8dec249-9f2a-4bfd-b83b-f72d50e5b936" (UID: "f8dec249-9f2a-4bfd-b83b-f72d50e5b936"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:11:13.855966 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.855747 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-workload-certs\") pod \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\" (UID: \"f8dec249-9f2a-4bfd-b83b-f72d50e5b936\") " Nov 03 21:11:13.855966 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.855918 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-credential-socket" pod "f8dec249-9f2a-4bfd-b83b-f72d50e5b936" (UID: "f8dec249-9f2a-4bfd-b83b-f72d50e5b936"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:11:13.856088 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.856045 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istiod-ca-cert" pod "f8dec249-9f2a-4bfd-b83b-f72d50e5b936" (UID: "f8dec249-9f2a-4bfd-b83b-f72d50e5b936"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:11:13.856088 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.856090 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-workload-certs" pod "f8dec249-9f2a-4bfd-b83b-f72d50e5b936" (UID: "f8dec249-9f2a-4bfd-b83b-f72d50e5b936"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:11:13.856383 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.856231 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:13.856383 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.856251 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:13.856383 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.856269 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:13.856383 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.856284 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:13.856383 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.856302 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:13.857781 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.857756 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-kube-api-access-z7w6r" pod "f8dec249-9f2a-4bfd-b83b-f72d50e5b936" (UID: "f8dec249-9f2a-4bfd-b83b-f72d50e5b936"). InnerVolumeSpecName "kube-api-access-z7w6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:11:13.857909 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.857821 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-token" pod "f8dec249-9f2a-4bfd-b83b-f72d50e5b936" (UID: "f8dec249-9f2a-4bfd-b83b-f72d50e5b936"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:11:13.857909 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.857876 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-podinfo" pod "f8dec249-9f2a-4bfd-b83b-f72d50e5b936" (UID: "f8dec249-9f2a-4bfd-b83b-f72d50e5b936"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:11:13.858000 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.857984 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-envoy" pod "f8dec249-9f2a-4bfd-b83b-f72d50e5b936" (UID: "f8dec249-9f2a-4bfd-b83b-f72d50e5b936"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:11:13.956996 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.956951 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7w6r\" (UniqueName: \"kubernetes.io/projected/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-kube-api-access-z7w6r\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:13.956996 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.956985 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:13.956996 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.956997 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:13.956996 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:13.957005 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8dec249-9f2a-4bfd-b83b-f72d50e5b936-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:14.457725 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:14.457689 4493 generic.go:356] "Generic (PLEG): container finished" podID="f8dec249-9f2a-4bfd-b83b-f72d50e5b936" containerID="742bad4705c31054cc620bed93507bf6120cc4755ecc17274c4da1d3e5a2e4de" exitCode=0 Nov 03 21:11:14.457896 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:14.457743 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" Nov 03 21:11:14.457896 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:14.457775 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" event={"ID":"f8dec249-9f2a-4bfd-b83b-f72d50e5b936","Type":"ContainerDied","Data":"742bad4705c31054cc620bed93507bf6120cc4755ecc17274c4da1d3e5a2e4de"} Nov 03 21:11:14.457896 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:14.457836 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r" event={"ID":"f8dec249-9f2a-4bfd-b83b-f72d50e5b936","Type":"ContainerDied","Data":"60809dd2298ffdb7153f93d64832e55c3dab201a92a7a5a5bef985831c5dc005"} Nov 03 21:11:14.457896 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:14.457866 4493 scope.go:122] "RemoveContainer" containerID="742bad4705c31054cc620bed93507bf6120cc4755ecc17274c4da1d3e5a2e4de" Nov 03 21:11:14.468208 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:14.468189 4493 scope.go:122] "RemoveContainer" containerID="742bad4705c31054cc620bed93507bf6120cc4755ecc17274c4da1d3e5a2e4de" Nov 03 21:11:14.468493 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:11:14.468474 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"742bad4705c31054cc620bed93507bf6120cc4755ecc17274c4da1d3e5a2e4de\": container with ID starting with 742bad4705c31054cc620bed93507bf6120cc4755ecc17274c4da1d3e5a2e4de not found: ID does not exist" containerID="742bad4705c31054cc620bed93507bf6120cc4755ecc17274c4da1d3e5a2e4de" Nov 03 21:11:14.468590 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:14.468504 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"742bad4705c31054cc620bed93507bf6120cc4755ecc17274c4da1d3e5a2e4de"} err="failed to get container status \"742bad4705c31054cc620bed93507bf6120cc4755ecc17274c4da1d3e5a2e4de\": rpc error: code = NotFound desc = could not find container \"742bad4705c31054cc620bed93507bf6120cc4755ecc17274c4da1d3e5a2e4de\": container with ID starting with 742bad4705c31054cc620bed93507bf6120cc4755ecc17274c4da1d3e5a2e4de not found: ID does not exist" Nov 03 21:11:14.629729 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:14.629702 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r"] Nov 03 21:11:14.688867 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:14.681912 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-qsp2r"] Nov 03 21:11:16.561817 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:16.561783 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8dec249-9f2a-4bfd-b83b-f72d50e5b936" path="/var/lib/kubelet/pods/f8dec249-9f2a-4bfd-b83b-f72d50e5b936/volumes" Nov 03 21:11:21.745086 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:11:21.745049 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:11:51.745062 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:11:51.745024 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:11:56.546829 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:56.546799 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4"] Nov 03 21:11:56.547223 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:11:56.547049 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" podUID="16823f97-8616-45c4-8d13-e05b7d4bf432" containerName="istio-proxy" containerID="cri-o://15f7f058d7f060996cb2be070b721d0e630dc09c2c1503ed553a27d99a047045" gracePeriod=30 Nov 03 21:12:01.736549 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:01.736513 4493 generic.go:356] "Generic (PLEG): container finished" podID="16823f97-8616-45c4-8d13-e05b7d4bf432" containerID="15f7f058d7f060996cb2be070b721d0e630dc09c2c1503ed553a27d99a047045" exitCode=0 Nov 03 21:12:01.736938 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:01.736590 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" event={"ID":"16823f97-8616-45c4-8d13-e05b7d4bf432","Type":"ContainerDied","Data":"15f7f058d7f060996cb2be070b721d0e630dc09c2c1503ed553a27d99a047045"} Nov 03 21:12:01.736938 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:01.736689 4493 scope.go:122] "RemoveContainer" containerID="c043fe533af5b8656516a7c85ade6f21fcc88348c5942b54e7aa8a3bb71674ce" Nov 03 21:12:01.805506 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:01.805483 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 21:12:01.923621 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:01.923525 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-token\" (UniqueName: \"kubernetes.io/projected/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-token\") pod \"16823f97-8616-45c4-8d13-e05b7d4bf432\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " Nov 03 21:12:01.923788 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:01.923636 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-workload-socket\") pod \"16823f97-8616-45c4-8d13-e05b7d4bf432\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " Nov 03 21:12:01.923788 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:01.923662 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16823f97-8616-45c4-8d13-e05b7d4bf432-kube-api-access-bjvxc\" (UniqueName: \"kubernetes.io/projected/16823f97-8616-45c4-8d13-e05b7d4bf432-kube-api-access-bjvxc\") pod \"16823f97-8616-45c4-8d13-e05b7d4bf432\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " Nov 03 21:12:01.923788 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:01.923697 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16823f97-8616-45c4-8d13-e05b7d4bf432-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16823f97-8616-45c4-8d13-e05b7d4bf432-istiod-ca-cert\") pod \"16823f97-8616-45c4-8d13-e05b7d4bf432\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " Nov 03 21:12:01.923788 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:01.923722 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-workload-certs\") pod \"16823f97-8616-45c4-8d13-e05b7d4bf432\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " Nov 03 21:12:01.924005 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:01.923794 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-credential-socket\") pod \"16823f97-8616-45c4-8d13-e05b7d4bf432\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " Nov 03 21:12:01.924005 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:01.923812 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-podinfo\") pod \"16823f97-8616-45c4-8d13-e05b7d4bf432\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " Nov 03 21:12:01.924005 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:01.923887 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-data\") pod \"16823f97-8616-45c4-8d13-e05b7d4bf432\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " Nov 03 21:12:01.924005 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:01.923940 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-envoy\") pod \"16823f97-8616-45c4-8d13-e05b7d4bf432\" (UID: \"16823f97-8616-45c4-8d13-e05b7d4bf432\") " Nov 03 21:12:01.924310 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:01.923924 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-workload-socket" pod "16823f97-8616-45c4-8d13-e05b7d4bf432" (UID: "16823f97-8616-45c4-8d13-e05b7d4bf432"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:12:01.924417 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:01.924217 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-credential-socket" pod "16823f97-8616-45c4-8d13-e05b7d4bf432" (UID: "16823f97-8616-45c4-8d13-e05b7d4bf432"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:12:01.924417 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:01.924258 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16823f97-8616-45c4-8d13-e05b7d4bf432-istiod-ca-cert" pod "16823f97-8616-45c4-8d13-e05b7d4bf432" (UID: "16823f97-8616-45c4-8d13-e05b7d4bf432"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:12:01.924417 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:01.924370 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-workload-certs" pod "16823f97-8616-45c4-8d13-e05b7d4bf432" (UID: "16823f97-8616-45c4-8d13-e05b7d4bf432"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:12:01.924562 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:01.924417 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-data" pod "16823f97-8616-45c4-8d13-e05b7d4bf432" (UID: "16823f97-8616-45c4-8d13-e05b7d4bf432"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:12:01.924562 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:01.924488 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:01.924562 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:01.924508 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16823f97-8616-45c4-8d13-e05b7d4bf432-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:01.924562 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:01.924523 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:01.924562 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:01.924537 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:01.926588 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:01.926559 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16823f97-8616-45c4-8d13-e05b7d4bf432-kube-api-access-bjvxc" pod "16823f97-8616-45c4-8d13-e05b7d4bf432" (UID: "16823f97-8616-45c4-8d13-e05b7d4bf432"). InnerVolumeSpecName "kube-api-access-bjvxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:12:01.926696 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:01.926589 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-token" pod "16823f97-8616-45c4-8d13-e05b7d4bf432" (UID: "16823f97-8616-45c4-8d13-e05b7d4bf432"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:12:01.926901 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:01.926881 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-podinfo" pod "16823f97-8616-45c4-8d13-e05b7d4bf432" (UID: "16823f97-8616-45c4-8d13-e05b7d4bf432"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:12:01.927134 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:01.927103 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-envoy" pod "16823f97-8616-45c4-8d13-e05b7d4bf432" (UID: "16823f97-8616-45c4-8d13-e05b7d4bf432"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:12:02.026119 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:02.026080 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:02.026119 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:02.026120 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:02.026318 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:02.026136 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:02.026318 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:02.026145 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16823f97-8616-45c4-8d13-e05b7d4bf432-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:02.026318 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:02.026156 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjvxc\" (UniqueName: \"kubernetes.io/projected/16823f97-8616-45c4-8d13-e05b7d4bf432-kube-api-access-bjvxc\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:02.744489 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:02.744449 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" event={"ID":"16823f97-8616-45c4-8d13-e05b7d4bf432","Type":"ContainerDied","Data":"cb36e66b3d63e7b8f94eed540320171e7d9afc85863c2c2793ff5bee2d4b1a39"} Nov 03 21:12:02.744961 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:02.744496 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4" Nov 03 21:12:02.744961 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:02.744548 4493 scope.go:122] "RemoveContainer" containerID="15f7f058d7f060996cb2be070b721d0e630dc09c2c1503ed553a27d99a047045" Nov 03 21:12:02.915890 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:02.915813 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4"] Nov 03 21:12:02.966508 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:02.966472 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-lxbl4"] Nov 03 21:12:04.562151 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:12:04.562117 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16823f97-8616-45c4-8d13-e05b7d4bf432" path="/var/lib/kubelet/pods/16823f97-8616-45c4-8d13-e05b7d4bf432/volumes" Nov 03 21:12:21.744487 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:12:21.744446 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:12:51.744733 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:12:51.744700 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:13:21.743976 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:13:21.743939 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:13:39.267536 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:39.267495 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n"] Nov 03 21:13:39.268174 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:39.267789 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n" podUID="167d7ad0-634f-467c-947e-58d1f1e69fe4" containerName="maas-api" containerID="cri-o://a30eec0fa49e7b5e1839931a33d928a1bfce56feb205a448c1627f25a642010e" gracePeriod=30 Nov 03 21:13:39.539882 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:39.539849 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n" Nov 03 21:13:39.586885 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:39.586855 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/167d7ad0-634f-467c-947e-58d1f1e69fe4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/167d7ad0-634f-467c-947e-58d1f1e69fe4-openshift-service-ca\") pod \"167d7ad0-634f-467c-947e-58d1f1e69fe4\" (UID: \"167d7ad0-634f-467c-947e-58d1f1e69fe4\") " Nov 03 21:13:39.587069 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:39.586924 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/167d7ad0-634f-467c-947e-58d1f1e69fe4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/167d7ad0-634f-467c-947e-58d1f1e69fe4-metrics-certs\") pod \"167d7ad0-634f-467c-947e-58d1f1e69fe4\" (UID: \"167d7ad0-634f-467c-947e-58d1f1e69fe4\") " Nov 03 21:13:39.587069 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:39.586968 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/167d7ad0-634f-467c-947e-58d1f1e69fe4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167d7ad0-634f-467c-947e-58d1f1e69fe4-maas-api-tls\") pod \"167d7ad0-634f-467c-947e-58d1f1e69fe4\" (UID: \"167d7ad0-634f-467c-947e-58d1f1e69fe4\") " Nov 03 21:13:39.587069 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:39.586988 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/167d7ad0-634f-467c-947e-58d1f1e69fe4-kube-api-access-4sj67\" (UniqueName: \"kubernetes.io/projected/167d7ad0-634f-467c-947e-58d1f1e69fe4-kube-api-access-4sj67\") pod \"167d7ad0-634f-467c-947e-58d1f1e69fe4\" (UID: \"167d7ad0-634f-467c-947e-58d1f1e69fe4\") " Nov 03 21:13:39.587294 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:39.587263 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/167d7ad0-634f-467c-947e-58d1f1e69fe4-openshift-service-ca" pod "167d7ad0-634f-467c-947e-58d1f1e69fe4" (UID: "167d7ad0-634f-467c-947e-58d1f1e69fe4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:39.587438 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:39.587418 4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/167d7ad0-634f-467c-947e-58d1f1e69fe4-openshift-service-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:39.589295 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:39.589268 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/167d7ad0-634f-467c-947e-58d1f1e69fe4-metrics-certs" pod "167d7ad0-634f-467c-947e-58d1f1e69fe4" (UID: "167d7ad0-634f-467c-947e-58d1f1e69fe4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:13:39.589446 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:39.589355 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/167d7ad0-634f-467c-947e-58d1f1e69fe4-kube-api-access-4sj67" pod "167d7ad0-634f-467c-947e-58d1f1e69fe4" (UID: "167d7ad0-634f-467c-947e-58d1f1e69fe4"). InnerVolumeSpecName "kube-api-access-4sj67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:39.589446 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:39.589376 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/167d7ad0-634f-467c-947e-58d1f1e69fe4-maas-api-tls" pod "167d7ad0-634f-467c-947e-58d1f1e69fe4" (UID: "167d7ad0-634f-467c-947e-58d1f1e69fe4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:13:39.688707 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:39.688668 4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/167d7ad0-634f-467c-947e-58d1f1e69fe4-metrics-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:39.688896 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:39.688842 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167d7ad0-634f-467c-947e-58d1f1e69fe4-maas-api-tls\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:39.688896 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:39.688865 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sj67\" (UniqueName: \"kubernetes.io/projected/167d7ad0-634f-467c-947e-58d1f1e69fe4-kube-api-access-4sj67\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:40.351708 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:40.351676 4493 generic.go:356] "Generic (PLEG): container finished" podID="167d7ad0-634f-467c-947e-58d1f1e69fe4" containerID="a30eec0fa49e7b5e1839931a33d928a1bfce56feb205a448c1627f25a642010e" exitCode=0 Nov 03 21:13:40.352120 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:40.351732 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n" Nov 03 21:13:40.352120 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:40.351745 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n" event={"ID":"167d7ad0-634f-467c-947e-58d1f1e69fe4","Type":"ContainerDied","Data":"a30eec0fa49e7b5e1839931a33d928a1bfce56feb205a448c1627f25a642010e"} Nov 03 21:13:40.352120 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:40.351806 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n" event={"ID":"167d7ad0-634f-467c-947e-58d1f1e69fe4","Type":"ContainerDied","Data":"bd3d80bbdddce72dcea4771adf015bf5ed1ac8dd0410e4a157edb9dac0979017"} Nov 03 21:13:40.352120 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:40.351835 4493 scope.go:122] "RemoveContainer" containerID="a30eec0fa49e7b5e1839931a33d928a1bfce56feb205a448c1627f25a642010e" Nov 03 21:13:40.361916 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:40.361898 4493 scope.go:122] "RemoveContainer" containerID="a30eec0fa49e7b5e1839931a33d928a1bfce56feb205a448c1627f25a642010e" Nov 03 21:13:40.362203 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:13:40.362185 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a30eec0fa49e7b5e1839931a33d928a1bfce56feb205a448c1627f25a642010e\": container with ID starting with a30eec0fa49e7b5e1839931a33d928a1bfce56feb205a448c1627f25a642010e not found: ID does not exist" containerID="a30eec0fa49e7b5e1839931a33d928a1bfce56feb205a448c1627f25a642010e" Nov 03 21:13:40.362278 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:40.362214 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a30eec0fa49e7b5e1839931a33d928a1bfce56feb205a448c1627f25a642010e"} err="failed to get container status \"a30eec0fa49e7b5e1839931a33d928a1bfce56feb205a448c1627f25a642010e\": rpc error: code = NotFound desc = could not find container \"a30eec0fa49e7b5e1839931a33d928a1bfce56feb205a448c1627f25a642010e\": container with ID starting with a30eec0fa49e7b5e1839931a33d928a1bfce56feb205a448c1627f25a642010e not found: ID does not exist" Nov 03 21:13:40.521161 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:40.521130 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n"] Nov 03 21:13:40.524740 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:40.524716 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-41aa21-67b9c58cb-mrl9n"] Nov 03 21:13:40.562026 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:40.561992 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="167d7ad0-634f-467c-947e-58d1f1e69fe4" path="/var/lib/kubelet/pods/167d7ad0-634f-467c-947e-58d1f1e69fe4/volumes" Nov 03 21:13:48.284205 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:48.284164 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf"] Nov 03 21:13:48.284699 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:48.284425 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" podUID="f1a56925-1c64-4165-abf5-3c9cc1cf22f3" containerName="istio-proxy" containerID="cri-o://e41c18a2406552fea87e4f7a67fe3c8ae6626b1c5014413781ba4eae7523091f" gracePeriod=30 Nov 03 21:13:49.068689 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:49.068657 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 21:13:49.071435 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:49.071409 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 21:13:51.746111 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:13:51.746070 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:13:53.427580 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.427545 4493 generic.go:356] "Generic (PLEG): container finished" podID="f1a56925-1c64-4165-abf5-3c9cc1cf22f3" containerID="e41c18a2406552fea87e4f7a67fe3c8ae6626b1c5014413781ba4eae7523091f" exitCode=0 Nov 03 21:13:53.428066 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.427627 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" event={"ID":"f1a56925-1c64-4165-abf5-3c9cc1cf22f3","Type":"ContainerDied","Data":"e41c18a2406552fea87e4f7a67fe3c8ae6626b1c5014413781ba4eae7523091f"} Nov 03 21:13:53.545156 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.545128 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 21:13:53.644961 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.644880 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-kube-api-access-8vf27\" (UniqueName: \"kubernetes.io/projected/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-kube-api-access-8vf27\") pod \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " Nov 03 21:13:53.644961 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.644927 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-envoy\") pod \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " Nov 03 21:13:53.645180 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.644992 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-token\" (UniqueName: \"kubernetes.io/projected/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-token\") pod \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " Nov 03 21:13:53.645180 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.645034 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-workload-socket\") pod \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " Nov 03 21:13:53.645180 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.645090 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istiod-ca-cert\") pod \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " Nov 03 21:13:53.645180 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.645124 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-podinfo\") pod \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " Nov 03 21:13:53.645180 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.645153 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-workload-certs\") pod \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " Nov 03 21:13:53.645911 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.645250 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-credential-socket\") pod \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " Nov 03 21:13:53.645911 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.645305 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-data\") pod \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\" (UID: \"f1a56925-1c64-4165-abf5-3c9cc1cf22f3\") " Nov 03 21:13:53.645911 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.645365 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-workload-socket" pod "f1a56925-1c64-4165-abf5-3c9cc1cf22f3" (UID: "f1a56925-1c64-4165-abf5-3c9cc1cf22f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:53.645911 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.645504 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istiod-ca-cert" pod "f1a56925-1c64-4165-abf5-3c9cc1cf22f3" (UID: "f1a56925-1c64-4165-abf5-3c9cc1cf22f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:53.645911 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.645836 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-data" pod "f1a56925-1c64-4165-abf5-3c9cc1cf22f3" (UID: "f1a56925-1c64-4165-abf5-3c9cc1cf22f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:53.646181 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.645952 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:53.646181 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.645974 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:53.646181 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.645989 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:53.646357 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.646182 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-workload-certs" pod "f1a56925-1c64-4165-abf5-3c9cc1cf22f3" (UID: "f1a56925-1c64-4165-abf5-3c9cc1cf22f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:53.646428 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.646396 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-credential-socket" pod "f1a56925-1c64-4165-abf5-3c9cc1cf22f3" (UID: "f1a56925-1c64-4165-abf5-3c9cc1cf22f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:53.647636 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.647597 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-kube-api-access-8vf27" pod "f1a56925-1c64-4165-abf5-3c9cc1cf22f3" (UID: "f1a56925-1c64-4165-abf5-3c9cc1cf22f3"). InnerVolumeSpecName "kube-api-access-8vf27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:53.647746 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.647689 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-envoy" pod "f1a56925-1c64-4165-abf5-3c9cc1cf22f3" (UID: "f1a56925-1c64-4165-abf5-3c9cc1cf22f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:53.647982 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.647966 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-podinfo" pod "f1a56925-1c64-4165-abf5-3c9cc1cf22f3" (UID: "f1a56925-1c64-4165-abf5-3c9cc1cf22f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:13:53.648630 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.648608 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-token" pod "f1a56925-1c64-4165-abf5-3c9cc1cf22f3" (UID: "f1a56925-1c64-4165-abf5-3c9cc1cf22f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:53.747714 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.747681 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:53.747714 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.747714 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:53.747937 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.747727 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:53.747937 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.747741 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vf27\" (UniqueName: \"kubernetes.io/projected/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-kube-api-access-8vf27\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:53.747937 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.747756 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:53.747937 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:53.747767 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1a56925-1c64-4165-abf5-3c9cc1cf22f3-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:54.434587 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:54.434551 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" event={"ID":"f1a56925-1c64-4165-abf5-3c9cc1cf22f3","Type":"ContainerDied","Data":"72d83c5663c3d8141d315f40690e9fb08c3a4fdc90bd122d59d61db405121308"} Nov 03 21:13:54.434587 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:54.434591 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" Nov 03 21:13:54.435093 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:54.434641 4493 scope.go:122] "RemoveContainer" containerID="e41c18a2406552fea87e4f7a67fe3c8ae6626b1c5014413781ba4eae7523091f" Nov 03 21:13:54.490530 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:54.490502 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:13:54.490645 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:54.490560 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf" podUID="f1a56925-1c64-4165-abf5-3c9cc1cf22f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:13:54.656405 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:54.656363 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf"] Nov 03 21:13:54.659218 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:54.659196 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-fqkjf"] Nov 03 21:13:56.561389 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:13:56.561358 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1a56925-1c64-4165-abf5-3c9cc1cf22f3" path="/var/lib/kubelet/pods/f1a56925-1c64-4165-abf5-3c9cc1cf22f3/volumes" Nov 03 21:14:21.744725 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:14:21.744680 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:14:51.750891 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:14:51.750848 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:15:21.744474 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:15:21.744436 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:15:36.276347 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:36.276305 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s"] Nov 03 21:15:36.385228 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:36.385198 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p"] Nov 03 21:15:39.942225 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:39.942195 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn"] Nov 03 21:15:39.942699 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:39.942404 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" podUID="7028d8a7-879e-4768-8bf2-390f98c23ac8" containerName="discovery" containerID="cri-o://d36ad86441a16fd2a0ca3e640d240b21a9719266b27eeff7c814e141ebd206c3" gracePeriod=30 Nov 03 21:15:40.219986 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:40.219956 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 21:15:40.317203 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:40.317175 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7028d8a7-879e-4768-8bf2-390f98c23ac8-kube-api-access-xj2rh\" (UniqueName: \"kubernetes.io/projected/7028d8a7-879e-4768-8bf2-390f98c23ac8-kube-api-access-xj2rh\") pod \"7028d8a7-879e-4768-8bf2-390f98c23ac8\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " Nov 03 21:15:40.317414 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:40.317227 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-csr-dns-cert\") pod \"7028d8a7-879e-4768-8bf2-390f98c23ac8\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " Nov 03 21:15:40.317414 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:40.317263 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-token\" (UniqueName: \"kubernetes.io/projected/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-token\") pod \"7028d8a7-879e-4768-8bf2-390f98c23ac8\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " Nov 03 21:15:40.317414 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:40.317282 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-kubeconfig\") pod \"7028d8a7-879e-4768-8bf2-390f98c23ac8\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " Nov 03 21:15:40.317414 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:40.317370 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7028d8a7-879e-4768-8bf2-390f98c23ac8-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7028d8a7-879e-4768-8bf2-390f98c23ac8-local-certs\") pod \"7028d8a7-879e-4768-8bf2-390f98c23ac8\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " Nov 03 21:15:40.317651 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:40.317491 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7028d8a7-879e-4768-8bf2-390f98c23ac8-cacerts\" (UniqueName: \"kubernetes.io/secret/7028d8a7-879e-4768-8bf2-390f98c23ac8-cacerts\") pod \"7028d8a7-879e-4768-8bf2-390f98c23ac8\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " Nov 03 21:15:40.317651 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:40.317540 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-csr-ca-configmap\") pod \"7028d8a7-879e-4768-8bf2-390f98c23ac8\" (UID: \"7028d8a7-879e-4768-8bf2-390f98c23ac8\") " Nov 03 21:15:40.318420 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:40.318318 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-csr-ca-configmap" pod "7028d8a7-879e-4768-8bf2-390f98c23ac8" (UID: "7028d8a7-879e-4768-8bf2-390f98c23ac8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:40.320042 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:40.320016 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7028d8a7-879e-4768-8bf2-390f98c23ac8-local-certs" pod "7028d8a7-879e-4768-8bf2-390f98c23ac8" (UID: "7028d8a7-879e-4768-8bf2-390f98c23ac8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:40.320134 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:40.320035 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-csr-dns-cert" pod "7028d8a7-879e-4768-8bf2-390f98c23ac8" (UID: "7028d8a7-879e-4768-8bf2-390f98c23ac8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:40.320268 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:40.320250 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7028d8a7-879e-4768-8bf2-390f98c23ac8-kube-api-access-xj2rh" pod "7028d8a7-879e-4768-8bf2-390f98c23ac8" (UID: "7028d8a7-879e-4768-8bf2-390f98c23ac8"). InnerVolumeSpecName "kube-api-access-xj2rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:40.320421 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:40.320396 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-token" pod "7028d8a7-879e-4768-8bf2-390f98c23ac8" (UID: "7028d8a7-879e-4768-8bf2-390f98c23ac8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:40.320421 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:40.320411 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7028d8a7-879e-4768-8bf2-390f98c23ac8-cacerts" pod "7028d8a7-879e-4768-8bf2-390f98c23ac8" (UID: "7028d8a7-879e-4768-8bf2-390f98c23ac8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:40.320559 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:40.320434 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-kubeconfig" pod "7028d8a7-879e-4768-8bf2-390f98c23ac8" (UID: "7028d8a7-879e-4768-8bf2-390f98c23ac8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:40.419411 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:40.419377 4493 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7028d8a7-879e-4768-8bf2-390f98c23ac8-cacerts\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:40.419411 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:40.419408 4493 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-csr-ca-configmap\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:40.419647 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:40.419423 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj2rh\" (UniqueName: \"kubernetes.io/projected/7028d8a7-879e-4768-8bf2-390f98c23ac8-kube-api-access-xj2rh\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:40.419647 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:40.419436 4493 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-csr-dns-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:40.419647 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:40.419449 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:40.419647 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:40.419460 4493 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7028d8a7-879e-4768-8bf2-390f98c23ac8-istio-kubeconfig\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:40.419647 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:40.419472 4493 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7028d8a7-879e-4768-8bf2-390f98c23ac8-local-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:41.043941 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:41.043847 4493 generic.go:356] "Generic (PLEG): container finished" podID="7028d8a7-879e-4768-8bf2-390f98c23ac8" containerID="d36ad86441a16fd2a0ca3e640d240b21a9719266b27eeff7c814e141ebd206c3" exitCode=0 Nov 03 21:15:41.043941 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:41.043918 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" Nov 03 21:15:41.044550 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:41.043934 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" event={"ID":"7028d8a7-879e-4768-8bf2-390f98c23ac8","Type":"ContainerDied","Data":"d36ad86441a16fd2a0ca3e640d240b21a9719266b27eeff7c814e141ebd206c3"} Nov 03 21:15:41.044550 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:41.044002 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn" event={"ID":"7028d8a7-879e-4768-8bf2-390f98c23ac8","Type":"ContainerDied","Data":"a06500fd3ebff2b4b22a525e4a23c798d0f2d8aa9350135b1b64f3a6d119a74f"} Nov 03 21:15:41.044550 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:41.044031 4493 scope.go:122] "RemoveContainer" containerID="d36ad86441a16fd2a0ca3e640d240b21a9719266b27eeff7c814e141ebd206c3" Nov 03 21:15:41.054320 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:41.054295 4493 scope.go:122] "RemoveContainer" containerID="d36ad86441a16fd2a0ca3e640d240b21a9719266b27eeff7c814e141ebd206c3" Nov 03 21:15:41.054739 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:15:41.054717 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d36ad86441a16fd2a0ca3e640d240b21a9719266b27eeff7c814e141ebd206c3\": container with ID starting with d36ad86441a16fd2a0ca3e640d240b21a9719266b27eeff7c814e141ebd206c3 not found: ID does not exist" containerID="d36ad86441a16fd2a0ca3e640d240b21a9719266b27eeff7c814e141ebd206c3" Nov 03 21:15:41.054831 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:41.054751 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d36ad86441a16fd2a0ca3e640d240b21a9719266b27eeff7c814e141ebd206c3"} err="failed to get container status \"d36ad86441a16fd2a0ca3e640d240b21a9719266b27eeff7c814e141ebd206c3\": rpc error: code = NotFound desc = could not find container \"d36ad86441a16fd2a0ca3e640d240b21a9719266b27eeff7c814e141ebd206c3\": container with ID starting with d36ad86441a16fd2a0ca3e640d240b21a9719266b27eeff7c814e141ebd206c3 not found: ID does not exist" Nov 03 21:15:41.075138 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:41.075111 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5"] Nov 03 21:15:41.075351 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:41.075303 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5" podUID="9f86bc65-604a-44f2-bbe5-8fdc21fa38a9" containerName="maas-api" containerID="cri-o://6afd8a22beb88a183351b7c1e357d9f63f93e27cba0cb313462bd0adf1f78ac6" gracePeriod=30 Nov 03 21:15:41.277049 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:41.277012 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s" podUID="1a6f7bb0-d357-43cb-9ad0-165987580c0b" containerName="payload-processing" containerID="cri-o://54152a273046453430849c4b0dfc5f3ac0c5a83d51d508e4464df5067d9872ed" gracePeriod=25 Nov 03 21:15:41.330825 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:41.330803 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5" Nov 03 21:15:41.359486 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:41.359456 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn"] Nov 03 21:15:41.363090 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:41.363070 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g4xjn"] Nov 03 21:15:41.386048 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:41.386016 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p" podUID="3cb2be7e-d668-49e9-82ac-3475659a928a" containerName="payload-pre-processing" containerID="cri-o://2b892d7c7369a314c6cd8056c7e8649ee4b772a78ddd547047a9add5fd86e43a" gracePeriod=25 Nov 03 21:15:41.429321 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:41.429294 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-openshift-service-ca\") pod \"9f86bc65-604a-44f2-bbe5-8fdc21fa38a9\" (UID: \"9f86bc65-604a-44f2-bbe5-8fdc21fa38a9\") " Nov 03 21:15:41.429468 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:41.429392 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-metrics-certs\") pod \"9f86bc65-604a-44f2-bbe5-8fdc21fa38a9\" (UID: \"9f86bc65-604a-44f2-bbe5-8fdc21fa38a9\") " Nov 03 21:15:41.429468 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:41.429421 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-kube-api-access-kv7ht\" (UniqueName: \"kubernetes.io/projected/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-kube-api-access-kv7ht\") pod \"9f86bc65-604a-44f2-bbe5-8fdc21fa38a9\" (UID: \"9f86bc65-604a-44f2-bbe5-8fdc21fa38a9\") " Nov 03 21:15:41.429539 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:41.429518 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-maas-api-tls\") pod \"9f86bc65-604a-44f2-bbe5-8fdc21fa38a9\" (UID: \"9f86bc65-604a-44f2-bbe5-8fdc21fa38a9\") " Nov 03 21:15:41.429653 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:41.429627 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-openshift-service-ca" pod "9f86bc65-604a-44f2-bbe5-8fdc21fa38a9" (UID: "9f86bc65-604a-44f2-bbe5-8fdc21fa38a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:41.430019 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:41.429949 4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-openshift-service-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:41.431646 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:41.431622 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-metrics-certs" pod "9f86bc65-604a-44f2-bbe5-8fdc21fa38a9" (UID: "9f86bc65-604a-44f2-bbe5-8fdc21fa38a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:41.431752 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:41.431705 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-kube-api-access-kv7ht" pod "9f86bc65-604a-44f2-bbe5-8fdc21fa38a9" (UID: "9f86bc65-604a-44f2-bbe5-8fdc21fa38a9"). InnerVolumeSpecName "kube-api-access-kv7ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:41.431793 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:41.431762 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-maas-api-tls" pod "9f86bc65-604a-44f2-bbe5-8fdc21fa38a9" (UID: "9f86bc65-604a-44f2-bbe5-8fdc21fa38a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:41.531325 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:41.531294 4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-metrics-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:41.531325 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:41.531321 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv7ht\" (UniqueName: \"kubernetes.io/projected/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-kube-api-access-kv7ht\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:41.531325 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:41.531356 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9-maas-api-tls\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:42.051582 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:42.051542 4493 generic.go:356] "Generic (PLEG): container finished" podID="9f86bc65-604a-44f2-bbe5-8fdc21fa38a9" containerID="6afd8a22beb88a183351b7c1e357d9f63f93e27cba0cb313462bd0adf1f78ac6" exitCode=0 Nov 03 21:15:42.052069 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:42.051593 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5" Nov 03 21:15:42.052069 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:42.051641 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5" event={"ID":"9f86bc65-604a-44f2-bbe5-8fdc21fa38a9","Type":"ContainerDied","Data":"6afd8a22beb88a183351b7c1e357d9f63f93e27cba0cb313462bd0adf1f78ac6"} Nov 03 21:15:42.052069 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:42.051702 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5" event={"ID":"9f86bc65-604a-44f2-bbe5-8fdc21fa38a9","Type":"ContainerDied","Data":"d8eb35e5d618489deaeb38a435c71bb1b46b0c8d470af8d081bd7515e3cf5a1a"} Nov 03 21:15:42.052069 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:42.051733 4493 scope.go:122] "RemoveContainer" containerID="6afd8a22beb88a183351b7c1e357d9f63f93e27cba0cb313462bd0adf1f78ac6" Nov 03 21:15:42.061998 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:42.061978 4493 scope.go:122] "RemoveContainer" containerID="6afd8a22beb88a183351b7c1e357d9f63f93e27cba0cb313462bd0adf1f78ac6" Nov 03 21:15:42.062259 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:15:42.062242 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6afd8a22beb88a183351b7c1e357d9f63f93e27cba0cb313462bd0adf1f78ac6\": container with ID starting with 6afd8a22beb88a183351b7c1e357d9f63f93e27cba0cb313462bd0adf1f78ac6 not found: ID does not exist" containerID="6afd8a22beb88a183351b7c1e357d9f63f93e27cba0cb313462bd0adf1f78ac6" Nov 03 21:15:42.062314 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:42.062268 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6afd8a22beb88a183351b7c1e357d9f63f93e27cba0cb313462bd0adf1f78ac6"} err="failed to get container status \"6afd8a22beb88a183351b7c1e357d9f63f93e27cba0cb313462bd0adf1f78ac6\": rpc error: code = NotFound desc = could not find container \"6afd8a22beb88a183351b7c1e357d9f63f93e27cba0cb313462bd0adf1f78ac6\": container with ID starting with 6afd8a22beb88a183351b7c1e357d9f63f93e27cba0cb313462bd0adf1f78ac6 not found: ID does not exist" Nov 03 21:15:42.220975 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:42.220942 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5"] Nov 03 21:15:42.272433 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:42.272405 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c14420-6c5c4d59d6-r78b5"] Nov 03 21:15:42.562156 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:42.562114 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7028d8a7-879e-4768-8bf2-390f98c23ac8" path="/var/lib/kubelet/pods/7028d8a7-879e-4768-8bf2-390f98c23ac8/volumes" Nov 03 21:15:42.562743 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:42.562722 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f86bc65-604a-44f2-bbe5-8fdc21fa38a9" path="/var/lib/kubelet/pods/9f86bc65-604a-44f2-bbe5-8fdc21fa38a9/volumes" Nov 03 21:15:44.782349 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:44.782279 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s" podUID="1a6f7bb0-d357-43cb-9ad0-165987580c0b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:15:44.782349 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:44.782318 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p" podUID="3cb2be7e-d668-49e9-82ac-3475659a928a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:15:47.292363 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:47.292304 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj"] Nov 03 21:15:47.292816 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:47.292540 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj" podUID="5a47f518-c3db-4150-99ee-00061ae2ae1e" containerName="maas-api" containerID="cri-o://21acb298c6d2242d7b6779a25dcfe2e2e3a5ee3c929ad238692ba96abb7a43a7" gracePeriod=30 Nov 03 21:15:47.556195 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:47.556172 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj" Nov 03 21:15:47.696606 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:47.696568 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a47f518-c3db-4150-99ee-00061ae2ae1e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a47f518-c3db-4150-99ee-00061ae2ae1e-metrics-certs\") pod \"5a47f518-c3db-4150-99ee-00061ae2ae1e\" (UID: \"5a47f518-c3db-4150-99ee-00061ae2ae1e\") " Nov 03 21:15:47.696606 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:47.696620 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a47f518-c3db-4150-99ee-00061ae2ae1e-kube-api-access-pm4x8\" (UniqueName: \"kubernetes.io/projected/5a47f518-c3db-4150-99ee-00061ae2ae1e-kube-api-access-pm4x8\") pod \"5a47f518-c3db-4150-99ee-00061ae2ae1e\" (UID: \"5a47f518-c3db-4150-99ee-00061ae2ae1e\") " Nov 03 21:15:47.696872 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:47.696760 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a47f518-c3db-4150-99ee-00061ae2ae1e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a47f518-c3db-4150-99ee-00061ae2ae1e-maas-api-tls\") pod \"5a47f518-c3db-4150-99ee-00061ae2ae1e\" (UID: \"5a47f518-c3db-4150-99ee-00061ae2ae1e\") " Nov 03 21:15:47.696937 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:47.696899 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a47f518-c3db-4150-99ee-00061ae2ae1e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a47f518-c3db-4150-99ee-00061ae2ae1e-openshift-service-ca\") pod \"5a47f518-c3db-4150-99ee-00061ae2ae1e\" (UID: \"5a47f518-c3db-4150-99ee-00061ae2ae1e\") " Nov 03 21:15:47.697650 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:47.697615 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a47f518-c3db-4150-99ee-00061ae2ae1e-openshift-service-ca" pod "5a47f518-c3db-4150-99ee-00061ae2ae1e" (UID: "5a47f518-c3db-4150-99ee-00061ae2ae1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:47.699937 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:47.699910 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a47f518-c3db-4150-99ee-00061ae2ae1e-maas-api-tls" pod "5a47f518-c3db-4150-99ee-00061ae2ae1e" (UID: "5a47f518-c3db-4150-99ee-00061ae2ae1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:47.699937 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:47.699930 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a47f518-c3db-4150-99ee-00061ae2ae1e-kube-api-access-pm4x8" pod "5a47f518-c3db-4150-99ee-00061ae2ae1e" (UID: "5a47f518-c3db-4150-99ee-00061ae2ae1e"). InnerVolumeSpecName "kube-api-access-pm4x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:47.700124 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:47.699939 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a47f518-c3db-4150-99ee-00061ae2ae1e-metrics-certs" pod "5a47f518-c3db-4150-99ee-00061ae2ae1e" (UID: "5a47f518-c3db-4150-99ee-00061ae2ae1e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:47.798807 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:47.798773 4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a47f518-c3db-4150-99ee-00061ae2ae1e-openshift-service-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:47.798807 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:47.798800 4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a47f518-c3db-4150-99ee-00061ae2ae1e-metrics-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:47.798807 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:47.798810 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm4x8\" (UniqueName: \"kubernetes.io/projected/5a47f518-c3db-4150-99ee-00061ae2ae1e-kube-api-access-pm4x8\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:47.799079 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:47.798820 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a47f518-c3db-4150-99ee-00061ae2ae1e-maas-api-tls\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:48.088424 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:48.088392 4493 generic.go:356] "Generic (PLEG): container finished" podID="5a47f518-c3db-4150-99ee-00061ae2ae1e" containerID="21acb298c6d2242d7b6779a25dcfe2e2e3a5ee3c929ad238692ba96abb7a43a7" exitCode=0 Nov 03 21:15:48.088608 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:48.088429 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj" event={"ID":"5a47f518-c3db-4150-99ee-00061ae2ae1e","Type":"ContainerDied","Data":"21acb298c6d2242d7b6779a25dcfe2e2e3a5ee3c929ad238692ba96abb7a43a7"} Nov 03 21:15:48.088608 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:48.088440 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj" Nov 03 21:15:48.088608 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:48.088469 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj" event={"ID":"5a47f518-c3db-4150-99ee-00061ae2ae1e","Type":"ContainerDied","Data":"b05b7bc63baa7df345fe1bfe6e1e6dcc3c72b3bab7edee804802c533cb27e874"} Nov 03 21:15:48.088608 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:48.088489 4493 scope.go:122] "RemoveContainer" containerID="21acb298c6d2242d7b6779a25dcfe2e2e3a5ee3c929ad238692ba96abb7a43a7" Nov 03 21:15:48.098887 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:48.098872 4493 scope.go:122] "RemoveContainer" containerID="21acb298c6d2242d7b6779a25dcfe2e2e3a5ee3c929ad238692ba96abb7a43a7" Nov 03 21:15:48.099150 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:15:48.099134 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21acb298c6d2242d7b6779a25dcfe2e2e3a5ee3c929ad238692ba96abb7a43a7\": container with ID starting with 21acb298c6d2242d7b6779a25dcfe2e2e3a5ee3c929ad238692ba96abb7a43a7 not found: ID does not exist" containerID="21acb298c6d2242d7b6779a25dcfe2e2e3a5ee3c929ad238692ba96abb7a43a7" Nov 03 21:15:48.099200 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:48.099156 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21acb298c6d2242d7b6779a25dcfe2e2e3a5ee3c929ad238692ba96abb7a43a7"} err="failed to get container status \"21acb298c6d2242d7b6779a25dcfe2e2e3a5ee3c929ad238692ba96abb7a43a7\": rpc error: code = NotFound desc = could not find container \"21acb298c6d2242d7b6779a25dcfe2e2e3a5ee3c929ad238692ba96abb7a43a7\": container with ID starting with 21acb298c6d2242d7b6779a25dcfe2e2e3a5ee3c929ad238692ba96abb7a43a7 not found: ID does not exist" Nov 03 21:15:48.257941 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:48.257911 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj"] Nov 03 21:15:48.261629 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:48.261606 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7c34ff-956c4c6cc-tl6fj"] Nov 03 21:15:48.563050 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:48.563022 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a47f518-c3db-4150-99ee-00061ae2ae1e" path="/var/lib/kubelet/pods/5a47f518-c3db-4150-99ee-00061ae2ae1e/volumes" Nov 03 21:15:50.199405 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:50.199373 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs"] Nov 03 21:15:50.199806 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:50.199593 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" podUID="b5b5e9e0-94df-40e7-b031-15c11babc655" containerName="istio-proxy" containerID="cri-o://34f3ccb6b6d54ea25a128595fb12b7d9d0fb465e14cd9b64fe59041ef4bbf1d9" gracePeriod=30 Nov 03 21:15:51.744778 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:15:51.744745 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:15:54.782581 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:54.782540 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p" podUID="3cb2be7e-d668-49e9-82ac-3475659a928a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:15:54.782979 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:54.782604 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s" podUID="1a6f7bb0-d357-43cb-9ad0-165987580c0b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:15:55.461491 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.461467 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:15:55.478702 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.478637 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-workload-socket\") pod \"b5b5e9e0-94df-40e7-b031-15c11babc655\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " Nov 03 21:15:55.478828 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.478728 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-token\" (UniqueName: \"kubernetes.io/projected/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-token\") pod \"b5b5e9e0-94df-40e7-b031-15c11babc655\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " Nov 03 21:15:55.478828 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.478762 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-credential-socket\") pod \"b5b5e9e0-94df-40e7-b031-15c11babc655\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " Nov 03 21:15:55.478828 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.478792 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-workload-certs\") pod \"b5b5e9e0-94df-40e7-b031-15c11babc655\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " Nov 03 21:15:55.479002 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.478902 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-workload-socket" pod "b5b5e9e0-94df-40e7-b031-15c11babc655" (UID: "b5b5e9e0-94df-40e7-b031-15c11babc655"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:55.479002 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.478942 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5b5e9e0-94df-40e7-b031-15c11babc655-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b5e9e0-94df-40e7-b031-15c11babc655-istiod-ca-cert\") pod \"b5b5e9e0-94df-40e7-b031-15c11babc655\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " Nov 03 21:15:55.479101 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.479013 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-envoy\") pod \"b5b5e9e0-94df-40e7-b031-15c11babc655\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " Nov 03 21:15:55.479101 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.479023 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-credential-socket" pod "b5b5e9e0-94df-40e7-b031-15c11babc655" (UID: "b5b5e9e0-94df-40e7-b031-15c11babc655"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:55.479101 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.479044 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-workload-certs" pod "b5b5e9e0-94df-40e7-b031-15c11babc655" (UID: "b5b5e9e0-94df-40e7-b031-15c11babc655"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:55.479227 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.479125 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5b5e9e0-94df-40e7-b031-15c11babc655-kube-api-access-2wfx2\" (UniqueName: \"kubernetes.io/projected/b5b5e9e0-94df-40e7-b031-15c11babc655-kube-api-access-2wfx2\") pod \"b5b5e9e0-94df-40e7-b031-15c11babc655\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " Nov 03 21:15:55.479227 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.479177 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-data\") pod \"b5b5e9e0-94df-40e7-b031-15c11babc655\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " Nov 03 21:15:55.479350 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.479233 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-podinfo\") pod \"b5b5e9e0-94df-40e7-b031-15c11babc655\" (UID: \"b5b5e9e0-94df-40e7-b031-15c11babc655\") " Nov 03 21:15:55.479350 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.479287 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b5e9e0-94df-40e7-b031-15c11babc655-istiod-ca-cert" pod "b5b5e9e0-94df-40e7-b031-15c11babc655" (UID: "b5b5e9e0-94df-40e7-b031-15c11babc655"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:55.479515 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.479479 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-data" pod "b5b5e9e0-94df-40e7-b031-15c11babc655" (UID: "b5b5e9e0-94df-40e7-b031-15c11babc655"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:55.479937 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.479878 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:55.479937 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.479905 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:55.479937 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.479924 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:55.479937 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.479937 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b5e9e0-94df-40e7-b031-15c11babc655-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:55.479937 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.479952 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:55.481234 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.481208 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-token" pod "b5b5e9e0-94df-40e7-b031-15c11babc655" (UID: "b5b5e9e0-94df-40e7-b031-15c11babc655"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:55.481368 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.481306 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-envoy" pod "b5b5e9e0-94df-40e7-b031-15c11babc655" (UID: "b5b5e9e0-94df-40e7-b031-15c11babc655"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:55.481763 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.481728 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-podinfo" pod "b5b5e9e0-94df-40e7-b031-15c11babc655" (UID: "b5b5e9e0-94df-40e7-b031-15c11babc655"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:15:55.481763 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.481745 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b5e9e0-94df-40e7-b031-15c11babc655-kube-api-access-2wfx2" pod "b5b5e9e0-94df-40e7-b031-15c11babc655" (UID: "b5b5e9e0-94df-40e7-b031-15c11babc655"). InnerVolumeSpecName "kube-api-access-2wfx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:55.580892 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.580861 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:55.580892 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.580885 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wfx2\" (UniqueName: \"kubernetes.io/projected/b5b5e9e0-94df-40e7-b031-15c11babc655-kube-api-access-2wfx2\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:55.580892 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.580897 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:55.581104 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:55.580905 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b5e9e0-94df-40e7-b031-15c11babc655-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:56.136956 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:56.136919 4493 generic.go:356] "Generic (PLEG): container finished" podID="b5b5e9e0-94df-40e7-b031-15c11babc655" containerID="34f3ccb6b6d54ea25a128595fb12b7d9d0fb465e14cd9b64fe59041ef4bbf1d9" exitCode=0 Nov 03 21:15:56.137424 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:56.136981 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" Nov 03 21:15:56.137424 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:56.137007 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" event={"ID":"b5b5e9e0-94df-40e7-b031-15c11babc655","Type":"ContainerDied","Data":"34f3ccb6b6d54ea25a128595fb12b7d9d0fb465e14cd9b64fe59041ef4bbf1d9"} Nov 03 21:15:56.137424 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:56.137063 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs" event={"ID":"b5b5e9e0-94df-40e7-b031-15c11babc655","Type":"ContainerDied","Data":"6730834c1a60dd1d7de0864ea8756bca7247754dc26cd4713e7b3c233d5f97f6"} Nov 03 21:15:56.137424 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:56.137086 4493 scope.go:122] "RemoveContainer" containerID="34f3ccb6b6d54ea25a128595fb12b7d9d0fb465e14cd9b64fe59041ef4bbf1d9" Nov 03 21:15:56.147115 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:56.147098 4493 scope.go:122] "RemoveContainer" containerID="ff8f9c1a71bb1c053be98b46bc6bb774a160468595af4f4fed0660ea97293553" Nov 03 21:15:56.156092 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:56.156073 4493 scope.go:122] "RemoveContainer" containerID="34f3ccb6b6d54ea25a128595fb12b7d9d0fb465e14cd9b64fe59041ef4bbf1d9" Nov 03 21:15:56.156361 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:15:56.156322 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34f3ccb6b6d54ea25a128595fb12b7d9d0fb465e14cd9b64fe59041ef4bbf1d9\": container with ID starting with 34f3ccb6b6d54ea25a128595fb12b7d9d0fb465e14cd9b64fe59041ef4bbf1d9 not found: ID does not exist" containerID="34f3ccb6b6d54ea25a128595fb12b7d9d0fb465e14cd9b64fe59041ef4bbf1d9" Nov 03 21:15:56.156415 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:56.156371 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34f3ccb6b6d54ea25a128595fb12b7d9d0fb465e14cd9b64fe59041ef4bbf1d9"} err="failed to get container status \"34f3ccb6b6d54ea25a128595fb12b7d9d0fb465e14cd9b64fe59041ef4bbf1d9\": rpc error: code = NotFound desc = could not find container \"34f3ccb6b6d54ea25a128595fb12b7d9d0fb465e14cd9b64fe59041ef4bbf1d9\": container with ID starting with 34f3ccb6b6d54ea25a128595fb12b7d9d0fb465e14cd9b64fe59041ef4bbf1d9 not found: ID does not exist" Nov 03 21:15:56.156415 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:56.156386 4493 scope.go:122] "RemoveContainer" containerID="ff8f9c1a71bb1c053be98b46bc6bb774a160468595af4f4fed0660ea97293553" Nov 03 21:15:56.156642 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:15:56.156622 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff8f9c1a71bb1c053be98b46bc6bb774a160468595af4f4fed0660ea97293553\": container with ID starting with ff8f9c1a71bb1c053be98b46bc6bb774a160468595af4f4fed0660ea97293553 not found: ID does not exist" containerID="ff8f9c1a71bb1c053be98b46bc6bb774a160468595af4f4fed0660ea97293553" Nov 03 21:15:56.156680 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:56.156649 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff8f9c1a71bb1c053be98b46bc6bb774a160468595af4f4fed0660ea97293553"} err="failed to get container status \"ff8f9c1a71bb1c053be98b46bc6bb774a160468595af4f4fed0660ea97293553\": rpc error: code = NotFound desc = could not find container \"ff8f9c1a71bb1c053be98b46bc6bb774a160468595af4f4fed0660ea97293553\": container with ID starting with ff8f9c1a71bb1c053be98b46bc6bb774a160468595af4f4fed0660ea97293553 not found: ID does not exist" Nov 03 21:15:56.309392 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:56.309361 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs"] Nov 03 21:15:56.312115 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:56.312095 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c14420-openshift-default-786c6cf5cf-plwbs"] Nov 03 21:15:56.561107 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:15:56.561076 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5b5e9e0-94df-40e7-b031-15c11babc655" path="/var/lib/kubelet/pods/b5b5e9e0-94df-40e7-b031-15c11babc655/volumes" Nov 03 21:16:01.540447 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:01.540423 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s" Nov 03 21:16:01.639603 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:01.639581 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p" Nov 03 21:16:01.650843 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:01.650822 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a6f7bb0-d357-43cb-9ad0-165987580c0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6f7bb0-d357-43cb-9ad0-165987580c0b-plugins-config-volume\") pod \"1a6f7bb0-d357-43cb-9ad0-165987580c0b\" (UID: \"1a6f7bb0-d357-43cb-9ad0-165987580c0b\") " Nov 03 21:16:01.650965 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:01.650908 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a6f7bb0-d357-43cb-9ad0-165987580c0b-kube-api-access-26hc4\" (UniqueName: \"kubernetes.io/projected/1a6f7bb0-d357-43cb-9ad0-165987580c0b-kube-api-access-26hc4\") pod \"1a6f7bb0-d357-43cb-9ad0-165987580c0b\" (UID: \"1a6f7bb0-d357-43cb-9ad0-165987580c0b\") " Nov 03 21:16:01.651190 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:01.651164 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a6f7bb0-d357-43cb-9ad0-165987580c0b-plugins-config-volume" pod "1a6f7bb0-d357-43cb-9ad0-165987580c0b" (UID: "1a6f7bb0-d357-43cb-9ad0-165987580c0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:01.651603 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:01.651573 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6f7bb0-d357-43cb-9ad0-165987580c0b-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:01.653083 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:01.653062 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a6f7bb0-d357-43cb-9ad0-165987580c0b-kube-api-access-26hc4" pod "1a6f7bb0-d357-43cb-9ad0-165987580c0b" (UID: "1a6f7bb0-d357-43cb-9ad0-165987580c0b"). InnerVolumeSpecName "kube-api-access-26hc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:01.752614 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:01.752582 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cb2be7e-d668-49e9-82ac-3475659a928a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb2be7e-d668-49e9-82ac-3475659a928a-plugins-config-volume\") pod \"3cb2be7e-d668-49e9-82ac-3475659a928a\" (UID: \"3cb2be7e-d668-49e9-82ac-3475659a928a\") " Nov 03 21:16:01.752791 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:01.752752 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb2be7e-d668-49e9-82ac-3475659a928a-kube-api-access-95fw4\" (UniqueName: \"kubernetes.io/projected/3cb2be7e-d668-49e9-82ac-3475659a928a-kube-api-access-95fw4\") pod \"3cb2be7e-d668-49e9-82ac-3475659a928a\" (UID: \"3cb2be7e-d668-49e9-82ac-3475659a928a\") " Nov 03 21:16:01.752978 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:01.752957 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb2be7e-d668-49e9-82ac-3475659a928a-plugins-config-volume" pod "3cb2be7e-d668-49e9-82ac-3475659a928a" (UID: "3cb2be7e-d668-49e9-82ac-3475659a928a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:01.753073 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:01.753059 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26hc4\" (UniqueName: \"kubernetes.io/projected/1a6f7bb0-d357-43cb-9ad0-165987580c0b-kube-api-access-26hc4\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:01.753112 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:01.753078 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb2be7e-d668-49e9-82ac-3475659a928a-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:01.754942 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:01.754919 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb2be7e-d668-49e9-82ac-3475659a928a-kube-api-access-95fw4" pod "3cb2be7e-d668-49e9-82ac-3475659a928a" (UID: "3cb2be7e-d668-49e9-82ac-3475659a928a"). InnerVolumeSpecName "kube-api-access-95fw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:01.853765 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:01.853690 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95fw4\" (UniqueName: \"kubernetes.io/projected/3cb2be7e-d668-49e9-82ac-3475659a928a-kube-api-access-95fw4\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:02.174034 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:02.173947 4493 generic.go:356] "Generic (PLEG): container finished" podID="3cb2be7e-d668-49e9-82ac-3475659a928a" containerID="2b892d7c7369a314c6cd8056c7e8649ee4b772a78ddd547047a9add5fd86e43a" exitCode=0 Nov 03 21:16:02.174034 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:02.174000 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p" Nov 03 21:16:02.174268 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:02.174028 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p" event={"ID":"3cb2be7e-d668-49e9-82ac-3475659a928a","Type":"ContainerDied","Data":"2b892d7c7369a314c6cd8056c7e8649ee4b772a78ddd547047a9add5fd86e43a"} Nov 03 21:16:02.174268 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:02.174080 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p" event={"ID":"3cb2be7e-d668-49e9-82ac-3475659a928a","Type":"ContainerDied","Data":"2c06eb354606d9da8f9ddc0bb8de52367ab9276a0def5874cec624d234f40941"} Nov 03 21:16:02.174268 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:02.174108 4493 scope.go:122] "RemoveContainer" containerID="2b892d7c7369a314c6cd8056c7e8649ee4b772a78ddd547047a9add5fd86e43a" Nov 03 21:16:02.175429 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:02.175402 4493 generic.go:356] "Generic (PLEG): container finished" podID="1a6f7bb0-d357-43cb-9ad0-165987580c0b" containerID="54152a273046453430849c4b0dfc5f3ac0c5a83d51d508e4464df5067d9872ed" exitCode=0 Nov 03 21:16:02.175556 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:02.175449 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s" Nov 03 21:16:02.175556 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:02.175484 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s" event={"ID":"1a6f7bb0-d357-43cb-9ad0-165987580c0b","Type":"ContainerDied","Data":"54152a273046453430849c4b0dfc5f3ac0c5a83d51d508e4464df5067d9872ed"} Nov 03 21:16:02.175556 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:02.175523 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s" event={"ID":"1a6f7bb0-d357-43cb-9ad0-165987580c0b","Type":"ContainerDied","Data":"f41560c93352fc4ef40f608106cbc0600b21d0047681ec213f539e63a5f54216"} Nov 03 21:16:02.183747 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:02.183731 4493 scope.go:122] "RemoveContainer" containerID="2b892d7c7369a314c6cd8056c7e8649ee4b772a78ddd547047a9add5fd86e43a" Nov 03 21:16:02.184037 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:16:02.184019 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b892d7c7369a314c6cd8056c7e8649ee4b772a78ddd547047a9add5fd86e43a\": container with ID starting with 2b892d7c7369a314c6cd8056c7e8649ee4b772a78ddd547047a9add5fd86e43a not found: ID does not exist" containerID="2b892d7c7369a314c6cd8056c7e8649ee4b772a78ddd547047a9add5fd86e43a" Nov 03 21:16:02.184093 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:02.184047 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b892d7c7369a314c6cd8056c7e8649ee4b772a78ddd547047a9add5fd86e43a"} err="failed to get container status \"2b892d7c7369a314c6cd8056c7e8649ee4b772a78ddd547047a9add5fd86e43a\": rpc error: code = NotFound desc = could not find container \"2b892d7c7369a314c6cd8056c7e8649ee4b772a78ddd547047a9add5fd86e43a\": container with ID starting with 2b892d7c7369a314c6cd8056c7e8649ee4b772a78ddd547047a9add5fd86e43a not found: ID does not exist" Nov 03 21:16:02.184093 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:02.184065 4493 scope.go:122] "RemoveContainer" containerID="54152a273046453430849c4b0dfc5f3ac0c5a83d51d508e4464df5067d9872ed" Nov 03 21:16:02.192844 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:02.192817 4493 scope.go:122] "RemoveContainer" containerID="54152a273046453430849c4b0dfc5f3ac0c5a83d51d508e4464df5067d9872ed" Nov 03 21:16:02.193092 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:16:02.193071 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54152a273046453430849c4b0dfc5f3ac0c5a83d51d508e4464df5067d9872ed\": container with ID starting with 54152a273046453430849c4b0dfc5f3ac0c5a83d51d508e4464df5067d9872ed not found: ID does not exist" containerID="54152a273046453430849c4b0dfc5f3ac0c5a83d51d508e4464df5067d9872ed" Nov 03 21:16:02.193159 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:02.193101 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54152a273046453430849c4b0dfc5f3ac0c5a83d51d508e4464df5067d9872ed"} err="failed to get container status \"54152a273046453430849c4b0dfc5f3ac0c5a83d51d508e4464df5067d9872ed\": rpc error: code = NotFound desc = could not find container \"54152a273046453430849c4b0dfc5f3ac0c5a83d51d508e4464df5067d9872ed\": container with ID starting with 54152a273046453430849c4b0dfc5f3ac0c5a83d51d508e4464df5067d9872ed not found: ID does not exist" Nov 03 21:16:02.342360 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:02.342315 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s"] Nov 03 21:16:02.394438 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:02.394396 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7c34ff-5cb77675ff-s4n8s"] Nov 03 21:16:02.460351 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:02.460250 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p"] Nov 03 21:16:02.463985 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:02.463951 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7c34ff-585fc85bc-ckp7p"] Nov 03 21:16:02.561071 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:02.561038 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a6f7bb0-d357-43cb-9ad0-165987580c0b" path="/var/lib/kubelet/pods/1a6f7bb0-d357-43cb-9ad0-165987580c0b/volumes" Nov 03 21:16:02.561468 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:02.561436 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cb2be7e-d668-49e9-82ac-3475659a928a" path="/var/lib/kubelet/pods/3cb2be7e-d668-49e9-82ac-3475659a928a/volumes" Nov 03 21:16:08.177981 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:08.177945 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9"] Nov 03 21:16:08.286097 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:08.286062 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858"] Nov 03 21:16:13.179264 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:13.179189 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9" podUID="956b284c-64a3-42f6-9fb6-20fd8b9f3d24" containerName="payload-processing" containerID="cri-o://6c7ef55fb3ea39691ec2572a7773d8b7a55f23ddda76a053a8ba4ad60c755a4f" gracePeriod=25 Nov 03 21:16:13.287102 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:13.287068 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858" podUID="0a905b64-861c-4a3b-94d3-89059db37170" containerName="payload-pre-processing" containerID="cri-o://2a9c03c3f5e2b8d3fc7e4a09e762e884e1fb669e701108a0784f9ce6f4cacc05" gracePeriod=25 Nov 03 21:16:13.318460 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:16:13.318432 4493 logging.go:55] [core] [Channel #2603 SubChannel #2605]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.139:50052", ServerName: "10.133.0.139:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.139:50052: connect: connection refused" Nov 03 21:16:13.319141 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:13.319108 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858" podUID="0a905b64-861c-4a3b-94d3-89059db37170" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:16:13.442971 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:13.442914 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9" Nov 03 21:16:13.569026 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:13.568987 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/956b284c-64a3-42f6-9fb6-20fd8b9f3d24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956b284c-64a3-42f6-9fb6-20fd8b9f3d24-plugins-config-volume\") pod \"956b284c-64a3-42f6-9fb6-20fd8b9f3d24\" (UID: \"956b284c-64a3-42f6-9fb6-20fd8b9f3d24\") " Nov 03 21:16:13.569217 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:13.569045 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/956b284c-64a3-42f6-9fb6-20fd8b9f3d24-kube-api-access-fpwtt\" (UniqueName: \"kubernetes.io/projected/956b284c-64a3-42f6-9fb6-20fd8b9f3d24-kube-api-access-fpwtt\") pod \"956b284c-64a3-42f6-9fb6-20fd8b9f3d24\" (UID: \"956b284c-64a3-42f6-9fb6-20fd8b9f3d24\") " Nov 03 21:16:13.569404 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:13.569380 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/956b284c-64a3-42f6-9fb6-20fd8b9f3d24-plugins-config-volume" pod "956b284c-64a3-42f6-9fb6-20fd8b9f3d24" (UID: "956b284c-64a3-42f6-9fb6-20fd8b9f3d24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:13.571442 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:13.571409 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/956b284c-64a3-42f6-9fb6-20fd8b9f3d24-kube-api-access-fpwtt" pod "956b284c-64a3-42f6-9fb6-20fd8b9f3d24" (UID: "956b284c-64a3-42f6-9fb6-20fd8b9f3d24"). InnerVolumeSpecName "kube-api-access-fpwtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:13.670171 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:13.670142 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956b284c-64a3-42f6-9fb6-20fd8b9f3d24-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:13.670171 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:13.670172 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpwtt\" (UniqueName: \"kubernetes.io/projected/956b284c-64a3-42f6-9fb6-20fd8b9f3d24-kube-api-access-fpwtt\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:14.243678 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:14.243643 4493 generic.go:356] "Generic (PLEG): container finished" podID="956b284c-64a3-42f6-9fb6-20fd8b9f3d24" containerID="6c7ef55fb3ea39691ec2572a7773d8b7a55f23ddda76a053a8ba4ad60c755a4f" exitCode=0 Nov 03 21:16:14.244192 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:14.243694 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9" event={"ID":"956b284c-64a3-42f6-9fb6-20fd8b9f3d24","Type":"ContainerDied","Data":"6c7ef55fb3ea39691ec2572a7773d8b7a55f23ddda76a053a8ba4ad60c755a4f"} Nov 03 21:16:14.244192 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:14.243702 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9" Nov 03 21:16:14.244192 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:14.243736 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9" event={"ID":"956b284c-64a3-42f6-9fb6-20fd8b9f3d24","Type":"ContainerDied","Data":"457f06cd3c41903c1e61624352d95062a44538b25316b7f96f630edf3dedd6c7"} Nov 03 21:16:14.244192 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:14.243757 4493 scope.go:122] "RemoveContainer" containerID="6c7ef55fb3ea39691ec2572a7773d8b7a55f23ddda76a053a8ba4ad60c755a4f" Nov 03 21:16:14.260066 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:14.260042 4493 scope.go:122] "RemoveContainer" containerID="6c7ef55fb3ea39691ec2572a7773d8b7a55f23ddda76a053a8ba4ad60c755a4f" Nov 03 21:16:14.260393 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:16:14.260369 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c7ef55fb3ea39691ec2572a7773d8b7a55f23ddda76a053a8ba4ad60c755a4f\": container with ID starting with 6c7ef55fb3ea39691ec2572a7773d8b7a55f23ddda76a053a8ba4ad60c755a4f not found: ID does not exist" containerID="6c7ef55fb3ea39691ec2572a7773d8b7a55f23ddda76a053a8ba4ad60c755a4f" Nov 03 21:16:14.260461 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:14.260402 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c7ef55fb3ea39691ec2572a7773d8b7a55f23ddda76a053a8ba4ad60c755a4f"} err="failed to get container status \"6c7ef55fb3ea39691ec2572a7773d8b7a55f23ddda76a053a8ba4ad60c755a4f\": rpc error: code = NotFound desc = could not find container \"6c7ef55fb3ea39691ec2572a7773d8b7a55f23ddda76a053a8ba4ad60c755a4f\": container with ID starting with 6c7ef55fb3ea39691ec2572a7773d8b7a55f23ddda76a053a8ba4ad60c755a4f not found: ID does not exist" Nov 03 21:16:14.319101 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:14.319057 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9" podUID="956b284c-64a3-42f6-9fb6-20fd8b9f3d24" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.139:50052\" within 1s: context deadline exceeded" Nov 03 21:16:14.413044 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:14.413008 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9"] Nov 03 21:16:14.416340 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:14.416305 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-98d1d2-79748fff9f-24ws9"] Nov 03 21:16:14.561280 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:14.561246 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="956b284c-64a3-42f6-9fb6-20fd8b9f3d24" path="/var/lib/kubelet/pods/956b284c-64a3-42f6-9fb6-20fd8b9f3d24/volumes" Nov 03 21:16:18.177316 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:18.177275 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7"] Nov 03 21:16:18.177741 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:18.177521 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7" podUID="b8d24cf1-1727-448d-bbad-94d523281bf2" containerName="maas-api" containerID="cri-o://1fdd43a8a1a31de1bb19da69a04a4de097228c5794785b7afd8ef846933a8bcf" gracePeriod=30 Nov 03 21:16:18.430100 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:18.430039 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7" Nov 03 21:16:18.522505 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:18.522476 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d24cf1-1727-448d-bbad-94d523281bf2-kube-api-access-tsjkl\" (UniqueName: \"kubernetes.io/projected/b8d24cf1-1727-448d-bbad-94d523281bf2-kube-api-access-tsjkl\") pod \"b8d24cf1-1727-448d-bbad-94d523281bf2\" (UID: \"b8d24cf1-1727-448d-bbad-94d523281bf2\") " Nov 03 21:16:18.522659 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:18.522565 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8d24cf1-1727-448d-bbad-94d523281bf2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d24cf1-1727-448d-bbad-94d523281bf2-openshift-service-ca\") pod \"b8d24cf1-1727-448d-bbad-94d523281bf2\" (UID: \"b8d24cf1-1727-448d-bbad-94d523281bf2\") " Nov 03 21:16:18.522659 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:18.522606 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8d24cf1-1727-448d-bbad-94d523281bf2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8d24cf1-1727-448d-bbad-94d523281bf2-maas-api-tls\") pod \"b8d24cf1-1727-448d-bbad-94d523281bf2\" (UID: \"b8d24cf1-1727-448d-bbad-94d523281bf2\") " Nov 03 21:16:18.522659 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:18.522647 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8d24cf1-1727-448d-bbad-94d523281bf2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d24cf1-1727-448d-bbad-94d523281bf2-metrics-certs\") pod \"b8d24cf1-1727-448d-bbad-94d523281bf2\" (UID: \"b8d24cf1-1727-448d-bbad-94d523281bf2\") " Nov 03 21:16:18.522986 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:18.522948 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d24cf1-1727-448d-bbad-94d523281bf2-openshift-service-ca" pod "b8d24cf1-1727-448d-bbad-94d523281bf2" (UID: "b8d24cf1-1727-448d-bbad-94d523281bf2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:18.524823 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:18.524797 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d24cf1-1727-448d-bbad-94d523281bf2-kube-api-access-tsjkl" pod "b8d24cf1-1727-448d-bbad-94d523281bf2" (UID: "b8d24cf1-1727-448d-bbad-94d523281bf2"). InnerVolumeSpecName "kube-api-access-tsjkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:18.524917 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:18.524823 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d24cf1-1727-448d-bbad-94d523281bf2-maas-api-tls" pod "b8d24cf1-1727-448d-bbad-94d523281bf2" (UID: "b8d24cf1-1727-448d-bbad-94d523281bf2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:16:18.524917 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:18.524871 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d24cf1-1727-448d-bbad-94d523281bf2-metrics-certs" pod "b8d24cf1-1727-448d-bbad-94d523281bf2" (UID: "b8d24cf1-1727-448d-bbad-94d523281bf2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:16:18.624181 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:18.624147 4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d24cf1-1727-448d-bbad-94d523281bf2-openshift-service-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:18.624181 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:18.624176 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8d24cf1-1727-448d-bbad-94d523281bf2-maas-api-tls\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:18.624181 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:18.624187 4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d24cf1-1727-448d-bbad-94d523281bf2-metrics-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:18.624472 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:18.624197 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsjkl\" (UniqueName: \"kubernetes.io/projected/b8d24cf1-1727-448d-bbad-94d523281bf2-kube-api-access-tsjkl\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:19.272400 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:19.272365 4493 generic.go:356] "Generic (PLEG): container finished" podID="b8d24cf1-1727-448d-bbad-94d523281bf2" containerID="1fdd43a8a1a31de1bb19da69a04a4de097228c5794785b7afd8ef846933a8bcf" exitCode=0 Nov 03 21:16:19.272881 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:19.272415 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7" Nov 03 21:16:19.272881 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:19.272449 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7" event={"ID":"b8d24cf1-1727-448d-bbad-94d523281bf2","Type":"ContainerDied","Data":"1fdd43a8a1a31de1bb19da69a04a4de097228c5794785b7afd8ef846933a8bcf"} Nov 03 21:16:19.272881 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:19.272505 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7" event={"ID":"b8d24cf1-1727-448d-bbad-94d523281bf2","Type":"ContainerDied","Data":"2a8e9578961b10436f8a24ce7759294d1ab284e56a222251a2375b39c2a943fe"} Nov 03 21:16:19.272881 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:19.272528 4493 scope.go:122] "RemoveContainer" containerID="1fdd43a8a1a31de1bb19da69a04a4de097228c5794785b7afd8ef846933a8bcf" Nov 03 21:16:19.281941 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:19.281922 4493 scope.go:122] "RemoveContainer" containerID="1fdd43a8a1a31de1bb19da69a04a4de097228c5794785b7afd8ef846933a8bcf" Nov 03 21:16:19.282212 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:16:19.282195 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fdd43a8a1a31de1bb19da69a04a4de097228c5794785b7afd8ef846933a8bcf\": container with ID starting with 1fdd43a8a1a31de1bb19da69a04a4de097228c5794785b7afd8ef846933a8bcf not found: ID does not exist" containerID="1fdd43a8a1a31de1bb19da69a04a4de097228c5794785b7afd8ef846933a8bcf" Nov 03 21:16:19.282264 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:19.282222 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fdd43a8a1a31de1bb19da69a04a4de097228c5794785b7afd8ef846933a8bcf"} err="failed to get container status \"1fdd43a8a1a31de1bb19da69a04a4de097228c5794785b7afd8ef846933a8bcf\": rpc error: code = NotFound desc = could not find container \"1fdd43a8a1a31de1bb19da69a04a4de097228c5794785b7afd8ef846933a8bcf\": container with ID starting with 1fdd43a8a1a31de1bb19da69a04a4de097228c5794785b7afd8ef846933a8bcf not found: ID does not exist" Nov 03 21:16:19.308164 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:19.308140 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r"] Nov 03 21:16:19.308405 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:19.308365 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r" podUID="bc018b8b-3821-4f83-b21a-9480a34aece0" containerName="maas-api" containerID="cri-o://9888a9899a9fef6a92dcde6990ca1d5b22a13870755315ab5d99c00bfe19dca0" gracePeriod=30 Nov 03 21:16:19.559938 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:19.559913 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r" Nov 03 21:16:19.590674 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:19.590644 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7"] Nov 03 21:16:19.592629 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:19.592610 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-562eb2-8674c6c5c7-84ss7"] Nov 03 21:16:19.635450 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:19.635419 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc018b8b-3821-4f83-b21a-9480a34aece0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc018b8b-3821-4f83-b21a-9480a34aece0-metrics-certs\") pod \"bc018b8b-3821-4f83-b21a-9480a34aece0\" (UID: \"bc018b8b-3821-4f83-b21a-9480a34aece0\") " Nov 03 21:16:19.635592 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:19.635455 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc018b8b-3821-4f83-b21a-9480a34aece0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc018b8b-3821-4f83-b21a-9480a34aece0-openshift-service-ca\") pod \"bc018b8b-3821-4f83-b21a-9480a34aece0\" (UID: \"bc018b8b-3821-4f83-b21a-9480a34aece0\") " Nov 03 21:16:19.635592 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:19.635513 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc018b8b-3821-4f83-b21a-9480a34aece0-kube-api-access-k5jct\" (UniqueName: \"kubernetes.io/projected/bc018b8b-3821-4f83-b21a-9480a34aece0-kube-api-access-k5jct\") pod \"bc018b8b-3821-4f83-b21a-9480a34aece0\" (UID: \"bc018b8b-3821-4f83-b21a-9480a34aece0\") " Nov 03 21:16:19.635592 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:19.635568 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc018b8b-3821-4f83-b21a-9480a34aece0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc018b8b-3821-4f83-b21a-9480a34aece0-maas-api-tls\") pod \"bc018b8b-3821-4f83-b21a-9480a34aece0\" (UID: \"bc018b8b-3821-4f83-b21a-9480a34aece0\") " Nov 03 21:16:19.635947 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:19.635913 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc018b8b-3821-4f83-b21a-9480a34aece0-openshift-service-ca" pod "bc018b8b-3821-4f83-b21a-9480a34aece0" (UID: "bc018b8b-3821-4f83-b21a-9480a34aece0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:19.636077 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:19.636063 4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc018b8b-3821-4f83-b21a-9480a34aece0-openshift-service-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:19.637898 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:19.637873 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc018b8b-3821-4f83-b21a-9480a34aece0-maas-api-tls" pod "bc018b8b-3821-4f83-b21a-9480a34aece0" (UID: "bc018b8b-3821-4f83-b21a-9480a34aece0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:16:19.637995 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:19.637925 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc018b8b-3821-4f83-b21a-9480a34aece0-kube-api-access-k5jct" pod "bc018b8b-3821-4f83-b21a-9480a34aece0" (UID: "bc018b8b-3821-4f83-b21a-9480a34aece0"). InnerVolumeSpecName "kube-api-access-k5jct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:19.638052 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:19.638011 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc018b8b-3821-4f83-b21a-9480a34aece0-metrics-certs" pod "bc018b8b-3821-4f83-b21a-9480a34aece0" (UID: "bc018b8b-3821-4f83-b21a-9480a34aece0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:16:19.737071 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:19.737038 4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc018b8b-3821-4f83-b21a-9480a34aece0-metrics-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:19.737071 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:19.737064 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5jct\" (UniqueName: \"kubernetes.io/projected/bc018b8b-3821-4f83-b21a-9480a34aece0-kube-api-access-k5jct\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:19.737071 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:19.737078 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc018b8b-3821-4f83-b21a-9480a34aece0-maas-api-tls\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:20.280050 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:20.280017 4493 generic.go:356] "Generic (PLEG): container finished" podID="bc018b8b-3821-4f83-b21a-9480a34aece0" containerID="9888a9899a9fef6a92dcde6990ca1d5b22a13870755315ab5d99c00bfe19dca0" exitCode=0 Nov 03 21:16:20.280528 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:20.280051 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r" event={"ID":"bc018b8b-3821-4f83-b21a-9480a34aece0","Type":"ContainerDied","Data":"9888a9899a9fef6a92dcde6990ca1d5b22a13870755315ab5d99c00bfe19dca0"} Nov 03 21:16:20.280528 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:20.280061 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r" Nov 03 21:16:20.280528 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:20.280106 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r" event={"ID":"bc018b8b-3821-4f83-b21a-9480a34aece0","Type":"ContainerDied","Data":"2794c9cf1ac54aacc9922d742d418fc18f0811198204f2115d87a48d83db2881"} Nov 03 21:16:20.280528 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:20.280133 4493 scope.go:122] "RemoveContainer" containerID="9888a9899a9fef6a92dcde6990ca1d5b22a13870755315ab5d99c00bfe19dca0" Nov 03 21:16:20.290734 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:20.290716 4493 scope.go:122] "RemoveContainer" containerID="9888a9899a9fef6a92dcde6990ca1d5b22a13870755315ab5d99c00bfe19dca0" Nov 03 21:16:20.290996 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:16:20.290979 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9888a9899a9fef6a92dcde6990ca1d5b22a13870755315ab5d99c00bfe19dca0\": container with ID starting with 9888a9899a9fef6a92dcde6990ca1d5b22a13870755315ab5d99c00bfe19dca0 not found: ID does not exist" containerID="9888a9899a9fef6a92dcde6990ca1d5b22a13870755315ab5d99c00bfe19dca0" Nov 03 21:16:20.291069 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:20.291004 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9888a9899a9fef6a92dcde6990ca1d5b22a13870755315ab5d99c00bfe19dca0"} err="failed to get container status \"9888a9899a9fef6a92dcde6990ca1d5b22a13870755315ab5d99c00bfe19dca0\": rpc error: code = NotFound desc = could not find container \"9888a9899a9fef6a92dcde6990ca1d5b22a13870755315ab5d99c00bfe19dca0\": container with ID starting with 9888a9899a9fef6a92dcde6990ca1d5b22a13870755315ab5d99c00bfe19dca0 not found: ID does not exist" Nov 03 21:16:20.449423 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:20.449394 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r"] Nov 03 21:16:20.451469 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:20.451448 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-98d1d2-67f644f4cf-t922r"] Nov 03 21:16:20.561018 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:20.560983 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8d24cf1-1727-448d-bbad-94d523281bf2" path="/var/lib/kubelet/pods/b8d24cf1-1727-448d-bbad-94d523281bf2/volumes" Nov 03 21:16:20.561574 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:20.561552 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc018b8b-3821-4f83-b21a-9480a34aece0" path="/var/lib/kubelet/pods/bc018b8b-3821-4f83-b21a-9480a34aece0/volumes" Nov 03 21:16:21.744495 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:16:21.744456 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:16:23.319093 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:23.319057 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858" podUID="0a905b64-861c-4a3b-94d3-89059db37170" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:16:33.318362 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:16:33.318324 4493 logging.go:55] [core] [Channel #2623 SubChannel #2624]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.138:50052", ServerName: "10.133.0.138:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.138:50052: connect: connection refused" Nov 03 21:16:33.547587 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:33.547565 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858" Nov 03 21:16:33.607679 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:33.607647 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a905b64-861c-4a3b-94d3-89059db37170-kube-api-access-p8zvb\" (UniqueName: \"kubernetes.io/projected/0a905b64-861c-4a3b-94d3-89059db37170-kube-api-access-p8zvb\") pod \"0a905b64-861c-4a3b-94d3-89059db37170\" (UID: \"0a905b64-861c-4a3b-94d3-89059db37170\") " Nov 03 21:16:33.607835 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:33.607726 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a905b64-861c-4a3b-94d3-89059db37170-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a905b64-861c-4a3b-94d3-89059db37170-plugins-config-volume\") pod \"0a905b64-861c-4a3b-94d3-89059db37170\" (UID: \"0a905b64-861c-4a3b-94d3-89059db37170\") " Nov 03 21:16:33.608105 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:33.608082 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a905b64-861c-4a3b-94d3-89059db37170-plugins-config-volume" pod "0a905b64-861c-4a3b-94d3-89059db37170" (UID: "0a905b64-861c-4a3b-94d3-89059db37170"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:33.609966 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:33.609942 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a905b64-861c-4a3b-94d3-89059db37170-kube-api-access-p8zvb" pod "0a905b64-861c-4a3b-94d3-89059db37170" (UID: "0a905b64-861c-4a3b-94d3-89059db37170"). InnerVolumeSpecName "kube-api-access-p8zvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:33.708971 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:33.708944 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8zvb\" (UniqueName: \"kubernetes.io/projected/0a905b64-861c-4a3b-94d3-89059db37170-kube-api-access-p8zvb\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:33.708971 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:33.708968 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a905b64-861c-4a3b-94d3-89059db37170-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:34.257842 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:34.257809 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm"] Nov 03 21:16:34.282777 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:34.282745 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv"] Nov 03 21:16:34.319099 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:34.319066 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858" podUID="0a905b64-861c-4a3b-94d3-89059db37170" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.138:50052\" within 1s: context deadline exceeded" Nov 03 21:16:34.319838 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:34.319145 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858" Nov 03 21:16:34.356536 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:34.356506 4493 generic.go:356] "Generic (PLEG): container finished" podID="0a905b64-861c-4a3b-94d3-89059db37170" containerID="2a9c03c3f5e2b8d3fc7e4a09e762e884e1fb669e701108a0784f9ce6f4cacc05" exitCode=0 Nov 03 21:16:34.356715 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:34.356557 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858" event={"ID":"0a905b64-861c-4a3b-94d3-89059db37170","Type":"ContainerDied","Data":"2a9c03c3f5e2b8d3fc7e4a09e762e884e1fb669e701108a0784f9ce6f4cacc05"} Nov 03 21:16:34.356715 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:34.356569 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858" Nov 03 21:16:34.356715 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:34.356608 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858" event={"ID":"0a905b64-861c-4a3b-94d3-89059db37170","Type":"ContainerDied","Data":"8c9981c20d4cd76d39312bf065bd097886cdbbd3ef9fdcc5befec9d3b0215837"} Nov 03 21:16:34.356715 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:34.356637 4493 scope.go:122] "RemoveContainer" containerID="2a9c03c3f5e2b8d3fc7e4a09e762e884e1fb669e701108a0784f9ce6f4cacc05" Nov 03 21:16:34.366414 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:34.366396 4493 scope.go:122] "RemoveContainer" containerID="2a9c03c3f5e2b8d3fc7e4a09e762e884e1fb669e701108a0784f9ce6f4cacc05" Nov 03 21:16:34.366693 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:16:34.366672 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a9c03c3f5e2b8d3fc7e4a09e762e884e1fb669e701108a0784f9ce6f4cacc05\": container with ID starting with 2a9c03c3f5e2b8d3fc7e4a09e762e884e1fb669e701108a0784f9ce6f4cacc05 not found: ID does not exist" containerID="2a9c03c3f5e2b8d3fc7e4a09e762e884e1fb669e701108a0784f9ce6f4cacc05" Nov 03 21:16:34.366772 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:34.366700 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a9c03c3f5e2b8d3fc7e4a09e762e884e1fb669e701108a0784f9ce6f4cacc05"} err="failed to get container status \"2a9c03c3f5e2b8d3fc7e4a09e762e884e1fb669e701108a0784f9ce6f4cacc05\": rpc error: code = NotFound desc = could not find container \"2a9c03c3f5e2b8d3fc7e4a09e762e884e1fb669e701108a0784f9ce6f4cacc05\": container with ID starting with 2a9c03c3f5e2b8d3fc7e4a09e762e884e1fb669e701108a0784f9ce6f4cacc05 not found: ID does not exist" Nov 03 21:16:34.489061 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:34.489031 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858"] Nov 03 21:16:34.491054 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:34.491034 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-98d1d2-5f455b5975-s9858"] Nov 03 21:16:34.560726 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:34.560701 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a905b64-861c-4a3b-94d3-89059db37170" path="/var/lib/kubelet/pods/0a905b64-861c-4a3b-94d3-89059db37170/volumes" Nov 03 21:16:49.278702 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.278655 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" containerID="cri-o://48bab4f0f58ac15df0d057dbf3c8d46688cb7c8a79ea2b1f67141164e4f53b2e" gracePeriod=45 Nov 03 21:16:49.303067 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.303033 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" containerID="cri-o://174a0686fb13ccea42cff9fedd4ad081969853f31f6922ad9f961088364a7f14" gracePeriod=45 Nov 03 21:16:49.442551 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.442521 4493 generic.go:356] "Generic (PLEG): container finished" podID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerID="48bab4f0f58ac15df0d057dbf3c8d46688cb7c8a79ea2b1f67141164e4f53b2e" exitCode=0 Nov 03 21:16:49.442715 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.442683 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" event={"ID":"2e9a83e5-1798-4ac7-9675-99d0caa9cde6","Type":"ContainerDied","Data":"48bab4f0f58ac15df0d057dbf3c8d46688cb7c8a79ea2b1f67141164e4f53b2e"} Nov 03 21:16:49.444277 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.444239 4493 generic.go:356] "Generic (PLEG): container finished" podID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerID="174a0686fb13ccea42cff9fedd4ad081969853f31f6922ad9f961088364a7f14" exitCode=0 Nov 03 21:16:49.444444 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.444302 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" event={"ID":"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1","Type":"ContainerDied","Data":"174a0686fb13ccea42cff9fedd4ad081969853f31f6922ad9f961088364a7f14"} Nov 03 21:16:49.559508 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.559484 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 21:16:49.578473 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.578450 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 21:16:49.674675 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.674641 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-tls-certs\") pod \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\" (UID: \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\") " Nov 03 21:16:49.674870 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.674704 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-home\" (UniqueName: \"kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-home\") pod \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\" (UID: \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\") " Nov 03 21:16:49.674870 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.674738 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-dshm\") pod \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\" (UID: \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\") " Nov 03 21:16:49.674870 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.674770 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-home\" (UniqueName: \"kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-home\") pod \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\" (UID: \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\") " Nov 03 21:16:49.674870 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.674808 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-model-cache\") pod \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\" (UID: \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\") " Nov 03 21:16:49.674870 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.674853 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-tls-certs\") pod \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\" (UID: \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\") " Nov 03 21:16:49.675158 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.674890 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-model-cache\") pod \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\" (UID: \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\") " Nov 03 21:16:49.675158 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.674925 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-tmp-dir\") pod \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\" (UID: \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\") " Nov 03 21:16:49.675158 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.674972 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-tmp-dir\") pod \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\" (UID: \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\") " Nov 03 21:16:49.675158 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.675004 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-kube-api-access-wmqmh\" (UniqueName: \"kubernetes.io/projected/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-kube-api-access-wmqmh\") pod \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\" (UID: \"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1\") " Nov 03 21:16:49.675158 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.675066 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-dshm\") pod \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\" (UID: \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\") " Nov 03 21:16:49.675158 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.675065 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-home" pod "2e9a83e5-1798-4ac7-9675-99d0caa9cde6" (UID: "2e9a83e5-1798-4ac7-9675-99d0caa9cde6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:49.675158 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.675154 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-kube-api-access-b9k62\" (UniqueName: \"kubernetes.io/projected/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-kube-api-access-b9k62\") pod \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\" (UID: \"2e9a83e5-1798-4ac7-9675-99d0caa9cde6\") " Nov 03 21:16:49.679532 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.675821 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-model-cache" pod "2e9a83e5-1798-4ac7-9675-99d0caa9cde6" (UID: "2e9a83e5-1798-4ac7-9675-99d0caa9cde6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:49.679532 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.675829 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-home" pod "a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" (UID: "a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:49.679532 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.675892 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-tmp-dir" pod "2e9a83e5-1798-4ac7-9675-99d0caa9cde6" (UID: "2e9a83e5-1798-4ac7-9675-99d0caa9cde6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:49.679532 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.675911 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-tmp-dir" pod "a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" (UID: "a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:49.679532 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.675961 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-model-cache" pod "a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" (UID: "a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:49.679532 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.678148 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-tls-certs" pod "a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" (UID: "a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:16:49.679532 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.678454 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-dshm" pod "2e9a83e5-1798-4ac7-9675-99d0caa9cde6" (UID: "2e9a83e5-1798-4ac7-9675-99d0caa9cde6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:49.679532 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.678552 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-kube-api-access-wmqmh" pod "a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" (UID: "a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1"). InnerVolumeSpecName "kube-api-access-wmqmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:49.679532 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.679072 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-kube-api-access-b9k62" pod "2e9a83e5-1798-4ac7-9675-99d0caa9cde6" (UID: "2e9a83e5-1798-4ac7-9675-99d0caa9cde6"). InnerVolumeSpecName "kube-api-access-b9k62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:49.679932 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.679571 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-tmp-dir\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:49.679932 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.679605 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmqmh\" (UniqueName: \"kubernetes.io/projected/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-kube-api-access-wmqmh\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:49.679932 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.679621 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-dshm\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:49.679932 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.679635 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9k62\" (UniqueName: \"kubernetes.io/projected/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-kube-api-access-b9k62\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:49.679932 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.679649 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-tls-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:49.679932 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.679663 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-home\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:49.679932 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.679676 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-home\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:49.679932 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.679690 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-model-cache\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:49.679932 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.679704 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-model-cache\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:49.679932 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.679717 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-tmp-dir\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:49.682273 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.682253 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-tls-certs" pod "2e9a83e5-1798-4ac7-9675-99d0caa9cde6" (UID: "2e9a83e5-1798-4ac7-9675-99d0caa9cde6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:16:49.683518 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.683497 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-dshm" pod "a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" (UID: "a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:49.780674 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.780592 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e9a83e5-1798-4ac7-9675-99d0caa9cde6-tls-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:49.780674 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:49.780630 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1-dshm\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:50.355280 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:50.355249 4493 scope.go:122] "RemoveContainer" containerID="48bab4f0f58ac15df0d057dbf3c8d46688cb7c8a79ea2b1f67141164e4f53b2e" Nov 03 21:16:50.364416 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:50.364393 4493 scope.go:122] "RemoveContainer" containerID="174a0686fb13ccea42cff9fedd4ad081969853f31f6922ad9f961088364a7f14" Nov 03 21:16:50.448315 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:50.448281 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" event={"ID":"a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1","Type":"ContainerDied","Data":"573c99a6b9db4d0b22fe0e8c340118fe35f2885826b703fb7008a80716531f21"} Nov 03 21:16:50.448455 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:50.448322 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv" Nov 03 21:16:50.448455 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:50.448362 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" event={"ID":"2e9a83e5-1798-4ac7-9675-99d0caa9cde6","Type":"ContainerDied","Data":"c1e588aa14558e8fc57d97e1f75b5dfb9bf9da0293998a73aaf04dde4d4207ba"} Nov 03 21:16:50.448455 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:50.448341 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm" Nov 03 21:16:50.714271 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:50.714237 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt"] Nov 03 21:16:50.714538 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:50.714488 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt" podUID="3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8" containerName="maas-api" containerID="cri-o://d426af98f29456e34761cadddbb23e28574a3a2087c8bdc276c67b74a3e3cf65" gracePeriod=30 Nov 03 21:16:50.795764 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:50.795734 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm"] Nov 03 21:16:50.799133 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:50.799102 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-984cbc/distinct-2-w4-984cbc-kserve-f9756bcb9-9klrm"] Nov 03 21:16:50.913082 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:50.912083 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv"] Nov 03 21:16:50.927988 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:50.927957 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-984cbc/simulator-w4-984cbc-kserve-5d8d94fbbb-pm4gv"] Nov 03 21:16:50.973673 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:50.973621 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt" Nov 03 21:16:51.093503 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:51.093465 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-metrics-certs\") pod \"3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8\" (UID: \"3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8\") " Nov 03 21:16:51.093696 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:51.093553 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-openshift-service-ca\") pod \"3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8\" (UID: \"3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8\") " Nov 03 21:16:51.093696 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:51.093582 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-kube-api-access-7rs62\" (UniqueName: \"kubernetes.io/projected/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-kube-api-access-7rs62\") pod \"3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8\" (UID: \"3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8\") " Nov 03 21:16:51.093696 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:51.093617 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-maas-api-tls\") pod \"3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8\" (UID: \"3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8\") " Nov 03 21:16:51.093975 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:51.093951 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-openshift-service-ca" pod "3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8" (UID: "3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:51.095905 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:51.095879 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-maas-api-tls" pod "3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8" (UID: "3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:16:51.096030 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:51.095942 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-metrics-certs" pod "3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8" (UID: "3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:16:51.096146 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:51.096048 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-kube-api-access-7rs62" pod "3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8" (UID: "3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8"). InnerVolumeSpecName "kube-api-access-7rs62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:51.194498 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:51.194467 4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-metrics-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:51.194498 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:51.194496 4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-openshift-service-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:51.194699 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:51.194506 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rs62\" (UniqueName: \"kubernetes.io/projected/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-kube-api-access-7rs62\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:51.194699 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:51.194516 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8-maas-api-tls\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:51.455308 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:51.455276 4493 generic.go:356] "Generic (PLEG): container finished" podID="3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8" containerID="d426af98f29456e34761cadddbb23e28574a3a2087c8bdc276c67b74a3e3cf65" exitCode=0 Nov 03 21:16:51.455308 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:51.455314 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt" event={"ID":"3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8","Type":"ContainerDied","Data":"d426af98f29456e34761cadddbb23e28574a3a2087c8bdc276c67b74a3e3cf65"} Nov 03 21:16:51.455921 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:51.455349 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt" Nov 03 21:16:51.455921 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:51.455375 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt" event={"ID":"3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8","Type":"ContainerDied","Data":"4401bfe082976c6bfb730d1a6ca7543ed9de31be931b90e19c1cb026a6eb4ee7"} Nov 03 21:16:51.455921 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:51.455398 4493 scope.go:122] "RemoveContainer" containerID="d426af98f29456e34761cadddbb23e28574a3a2087c8bdc276c67b74a3e3cf65" Nov 03 21:16:51.465514 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:51.465497 4493 scope.go:122] "RemoveContainer" containerID="d426af98f29456e34761cadddbb23e28574a3a2087c8bdc276c67b74a3e3cf65" Nov 03 21:16:51.465762 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:16:51.465741 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d426af98f29456e34761cadddbb23e28574a3a2087c8bdc276c67b74a3e3cf65\": container with ID starting with d426af98f29456e34761cadddbb23e28574a3a2087c8bdc276c67b74a3e3cf65 not found: ID does not exist" containerID="d426af98f29456e34761cadddbb23e28574a3a2087c8bdc276c67b74a3e3cf65" Nov 03 21:16:51.465858 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:51.465769 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d426af98f29456e34761cadddbb23e28574a3a2087c8bdc276c67b74a3e3cf65"} err="failed to get container status \"d426af98f29456e34761cadddbb23e28574a3a2087c8bdc276c67b74a3e3cf65\": rpc error: code = NotFound desc = could not find container \"d426af98f29456e34761cadddbb23e28574a3a2087c8bdc276c67b74a3e3cf65\": container with ID starting with d426af98f29456e34761cadddbb23e28574a3a2087c8bdc276c67b74a3e3cf65 not found: ID does not exist" Nov 03 21:16:51.624515 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:51.624484 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt"] Nov 03 21:16:51.627553 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:51.627532 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-45f792-775c67c548-ddqrt"] Nov 03 21:16:51.744244 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:16:51.744162 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:16:52.560892 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:52.560863 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" path="/var/lib/kubelet/pods/2e9a83e5-1798-4ac7-9675-99d0caa9cde6/volumes" Nov 03 21:16:52.561280 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:52.561251 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8" path="/var/lib/kubelet/pods/3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8/volumes" Nov 03 21:16:52.561631 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:16:52.561618 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" path="/var/lib/kubelet/pods/a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1/volumes" Nov 03 21:17:00.301987 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.301955 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z"] Nov 03 21:17:00.302497 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302378 4493 cpu_manager.go:414] "containerMap: removing container" podUID="fec5bf6e-548b-4ace-9533-f2f6837fa1a0" containerName="maas-api" Nov 03 21:17:00.302497 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302391 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fec5bf6e-548b-4ace-9533-f2f6837fa1a0" containerName="maas-api" Nov 03 21:17:00.302497 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302407 4493 cpu_manager.go:414] "containerMap: removing container" podUID="3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8" containerName="maas-api" Nov 03 21:17:00.302497 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302412 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8" containerName="maas-api" Nov 03 21:17:00.302497 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302425 4493 cpu_manager.go:414] "containerMap: removing container" podUID="f8dec249-9f2a-4bfd-b83b-f72d50e5b936" containerName="istio-proxy" Nov 03 21:17:00.302497 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302430 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8dec249-9f2a-4bfd-b83b-f72d50e5b936" containerName="istio-proxy" Nov 03 21:17:00.302497 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302439 4493 cpu_manager.go:414] "containerMap: removing container" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" Nov 03 21:17:00.302497 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302444 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" Nov 03 21:17:00.302497 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302451 4493 cpu_manager.go:414] "containerMap: removing container" podUID="0a905b64-861c-4a3b-94d3-89059db37170" containerName="payload-pre-processing" Nov 03 21:17:00.302497 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302456 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a905b64-861c-4a3b-94d3-89059db37170" containerName="payload-pre-processing" Nov 03 21:17:00.302497 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302464 4493 cpu_manager.go:414] "containerMap: removing container" podUID="3cb2be7e-d668-49e9-82ac-3475659a928a" containerName="payload-pre-processing" Nov 03 21:17:00.302497 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302468 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cb2be7e-d668-49e9-82ac-3475659a928a" containerName="payload-pre-processing" Nov 03 21:17:00.302497 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302475 4493 cpu_manager.go:414] "containerMap: removing container" podUID="16823f97-8616-45c4-8d13-e05b7d4bf432" containerName="istio-proxy" Nov 03 21:17:00.302497 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302479 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16823f97-8616-45c4-8d13-e05b7d4bf432" containerName="istio-proxy" Nov 03 21:17:00.302497 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302487 4493 cpu_manager.go:414] "containerMap: removing container" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" Nov 03 21:17:00.302497 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302490 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" Nov 03 21:17:00.302497 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302496 4493 cpu_manager.go:414] "containerMap: removing container" podUID="21488ac9-bfba-4bb5-ba76-8c76c7c9fa21" containerName="istio-proxy" Nov 03 21:17:00.302497 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302500 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21488ac9-bfba-4bb5-ba76-8c76c7c9fa21" containerName="istio-proxy" Nov 03 21:17:00.302497 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302507 4493 cpu_manager.go:414] "containerMap: removing container" podUID="956b284c-64a3-42f6-9fb6-20fd8b9f3d24" containerName="payload-processing" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302511 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="956b284c-64a3-42f6-9fb6-20fd8b9f3d24" containerName="payload-processing" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302517 4493 cpu_manager.go:414] "containerMap: removing container" podUID="b8d24cf1-1727-448d-bbad-94d523281bf2" containerName="maas-api" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302524 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d24cf1-1727-448d-bbad-94d523281bf2" containerName="maas-api" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302568 4493 cpu_manager.go:414] "containerMap: removing container" podUID="bc018b8b-3821-4f83-b21a-9480a34aece0" containerName="maas-api" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302573 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc018b8b-3821-4f83-b21a-9480a34aece0" containerName="maas-api" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302581 4493 cpu_manager.go:414] "containerMap: removing container" podUID="9f86bc65-604a-44f2-bbe5-8fdc21fa38a9" containerName="maas-api" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302586 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f86bc65-604a-44f2-bbe5-8fdc21fa38a9" containerName="maas-api" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302592 4493 cpu_manager.go:414] "containerMap: removing container" podUID="1a6f7bb0-d357-43cb-9ad0-165987580c0b" containerName="payload-processing" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302597 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a6f7bb0-d357-43cb-9ad0-165987580c0b" containerName="payload-processing" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302605 4493 cpu_manager.go:414] "containerMap: removing container" podUID="21488ac9-bfba-4bb5-ba76-8c76c7c9fa21" containerName="istio-proxy" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302609 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21488ac9-bfba-4bb5-ba76-8c76c7c9fa21" containerName="istio-proxy" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302615 4493 cpu_manager.go:414] "containerMap: removing container" podUID="5a47f518-c3db-4150-99ee-00061ae2ae1e" containerName="maas-api" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302618 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a47f518-c3db-4150-99ee-00061ae2ae1e" containerName="maas-api" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302624 4493 cpu_manager.go:414] "containerMap: removing container" podUID="16823f97-8616-45c4-8d13-e05b7d4bf432" containerName="istio-proxy" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302629 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16823f97-8616-45c4-8d13-e05b7d4bf432" containerName="istio-proxy" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302634 4493 cpu_manager.go:414] "containerMap: removing container" podUID="7028d8a7-879e-4768-8bf2-390f98c23ac8" containerName="discovery" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302638 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7028d8a7-879e-4768-8bf2-390f98c23ac8" containerName="discovery" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302647 4493 cpu_manager.go:414] "containerMap: removing container" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302651 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302656 4493 cpu_manager.go:414] "containerMap: removing container" podUID="7ec5039b-2ca6-4099-9919-12f4b42af9c7" containerName="istio-proxy" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302660 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec5039b-2ca6-4099-9919-12f4b42af9c7" containerName="istio-proxy" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302669 4493 cpu_manager.go:414] "containerMap: removing container" podUID="f1a56925-1c64-4165-abf5-3c9cc1cf22f3" containerName="istio-proxy" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302673 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1a56925-1c64-4165-abf5-3c9cc1cf22f3" containerName="istio-proxy" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302679 4493 cpu_manager.go:414] "containerMap: removing container" podUID="b5b5e9e0-94df-40e7-b031-15c11babc655" containerName="istio-proxy" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302683 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5b5e9e0-94df-40e7-b031-15c11babc655" containerName="istio-proxy" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302692 4493 cpu_manager.go:414] "containerMap: removing container" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302696 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302703 4493 cpu_manager.go:414] "containerMap: removing container" podUID="701f971e-5fbb-4f4b-81ec-1dfa66005128" containerName="istio-proxy" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302708 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="701f971e-5fbb-4f4b-81ec-1dfa66005128" containerName="istio-proxy" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302715 4493 cpu_manager.go:414] "containerMap: removing container" podUID="167d7ad0-634f-467c-947e-58d1f1e69fe4" containerName="maas-api" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302718 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="167d7ad0-634f-467c-947e-58d1f1e69fe4" containerName="maas-api" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302724 4493 cpu_manager.go:414] "containerMap: removing container" podUID="701f971e-5fbb-4f4b-81ec-1dfa66005128" containerName="istio-proxy" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302729 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="701f971e-5fbb-4f4b-81ec-1dfa66005128" containerName="istio-proxy" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302736 4493 cpu_manager.go:414] "containerMap: removing container" podUID="49c1d4d0-289d-4028-a3c5-b16bbeb87b40" containerName="istio-proxy" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302741 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c1d4d0-289d-4028-a3c5-b16bbeb87b40" containerName="istio-proxy" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302818 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5b5e9e0-94df-40e7-b031-15c11babc655" containerName="istio-proxy" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302828 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c1d4d0-289d-4028-a3c5-b16bbeb87b40" containerName="istio-proxy" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302835 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fec5bf6e-548b-4ace-9533-f2f6837fa1a0" containerName="maas-api" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302840 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b85c1d1-26ab-4496-8721-fcf0b4ff3ff8" containerName="maas-api" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302848 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8dec249-9f2a-4bfd-b83b-f72d50e5b936" containerName="istio-proxy" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302853 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="50477b11-be5c-4d14-b7ad-663dd674f82e" containerName="main" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302859 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a905b64-861c-4a3b-94d3-89059db37170" containerName="payload-pre-processing" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302865 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cb2be7e-d668-49e9-82ac-3475659a928a" containerName="payload-pre-processing" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302871 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="16823f97-8616-45c4-8d13-e05b7d4bf432" containerName="istio-proxy" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302876 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d84d7f0a-8ffa-4211-a9be-d73ab3853b40" containerName="main" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302882 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="21488ac9-bfba-4bb5-ba76-8c76c7c9fa21" containerName="istio-proxy" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302886 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="956b284c-64a3-42f6-9fb6-20fd8b9f3d24" containerName="payload-processing" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302891 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d24cf1-1727-448d-bbad-94d523281bf2" containerName="maas-api" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302896 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc018b8b-3821-4f83-b21a-9480a34aece0" containerName="maas-api" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302902 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f86bc65-604a-44f2-bbe5-8fdc21fa38a9" containerName="maas-api" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302907 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a6f7bb0-d357-43cb-9ad0-165987580c0b" containerName="payload-processing" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302913 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="21488ac9-bfba-4bb5-ba76-8c76c7c9fa21" containerName="istio-proxy" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302918 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a47f518-c3db-4150-99ee-00061ae2ae1e" containerName="maas-api" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302924 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="16823f97-8616-45c4-8d13-e05b7d4bf432" containerName="istio-proxy" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302928 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7028d8a7-879e-4768-8bf2-390f98c23ac8" containerName="discovery" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302933 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6b8c002-2e0a-4f9e-86f6-23adb6f17ec1" containerName="main" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302937 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec5039b-2ca6-4099-9919-12f4b42af9c7" containerName="istio-proxy" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302943 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1a56925-1c64-4165-abf5-3c9cc1cf22f3" containerName="istio-proxy" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302949 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e9a83e5-1798-4ac7-9675-99d0caa9cde6" containerName="main" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302954 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="701f971e-5fbb-4f4b-81ec-1dfa66005128" containerName="istio-proxy" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302959 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="167d7ad0-634f-467c-947e-58d1f1e69fe4" containerName="maas-api" Nov 03 21:17:00.303078 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.302963 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="701f971e-5fbb-4f4b-81ec-1dfa66005128" containerName="istio-proxy" Nov 03 21:17:00.306916 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.306899 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z" Nov 03 21:17:00.357611 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.357575 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z"] Nov 03 21:17:00.406832 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.406804 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:17:00.407165 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.407147 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\"" Nov 03 21:17:00.485894 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.485852 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z\" (UID: \"742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z" Nov 03 21:17:00.486075 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.485934 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbcwc\" (UniqueName: \"kubernetes.io/projected/742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68-kube-api-access-vbcwc\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z\" (UID: \"742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z" Nov 03 21:17:00.486075 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.486016 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z\" (UID: \"742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z" Nov 03 21:17:00.586892 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.586812 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z\" (UID: \"742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z" Nov 03 21:17:00.586892 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.586874 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbcwc\" (UniqueName: \"kubernetes.io/projected/742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68-kube-api-access-vbcwc\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z\" (UID: \"742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z" Nov 03 21:17:00.587130 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.586911 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z\" (UID: \"742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z" Nov 03 21:17:00.587475 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.587453 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z\" (UID: \"742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z" Nov 03 21:17:00.589598 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.589574 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z\" (UID: \"742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z" Nov 03 21:17:00.646659 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.646631 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbcwc\" (UniqueName: \"kubernetes.io/projected/742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68-kube-api-access-vbcwc\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z\" (UID: \"742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z" Nov 03 21:17:00.920487 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:00.920405 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z" Nov 03 21:17:01.300272 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:01.300229 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z"] Nov 03 21:17:01.301230 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:17:01.301204 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod742e26fe_9b4b_41a2_bb7b_3a62ff2b5e68.slice/crio-55c661d2e6c4401e32385380abc7f0dd00f93a4e7fb3dade52eee6ecf2241525 WatchSource:0}: Error finding container 55c661d2e6c4401e32385380abc7f0dd00f93a4e7fb3dade52eee6ecf2241525: Status 404 returned error can't find the container with id 55c661d2e6c4401e32385380abc7f0dd00f93a4e7fb3dade52eee6ecf2241525 Nov 03 21:17:01.509562 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:01.509530 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z" event={"ID":"742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68","Type":"ContainerStarted","Data":"55c661d2e6c4401e32385380abc7f0dd00f93a4e7fb3dade52eee6ecf2241525"} Nov 03 21:17:02.516948 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:02.516856 4493 generic.go:356] "Generic (PLEG): container finished" podID="742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68" containerID="605752abdbfe18c58f9674c970f85ded053946c95cf9de8a6d5e076d6e3d52f3" exitCode=0 Nov 03 21:17:02.517367 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:02.516936 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z" event={"ID":"742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68","Type":"ContainerDied","Data":"605752abdbfe18c58f9674c970f85ded053946c95cf9de8a6d5e076d6e3d52f3"} Nov 03 21:17:03.644506 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:03.644472 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z" Nov 03 21:17:03.723114 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:03.723091 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68-maas-api-xks-ca\") pod \"742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68\" (UID: \"742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68\") " Nov 03 21:17:03.723274 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:03.723229 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68-kube-api-access-vbcwc\" (UniqueName: \"kubernetes.io/projected/742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68-kube-api-access-vbcwc\") pod \"742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68\" (UID: \"742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68\") " Nov 03 21:17:03.723274 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:03.723267 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68-maas-api-openshift-ca\") pod \"742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68\" (UID: \"742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68\") " Nov 03 21:17:03.723710 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:03.723687 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68-maas-api-openshift-ca" pod "742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68" (UID: "742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:03.725215 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:03.725195 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68-maas-api-xks-ca" pod "742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68" (UID: "742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:17:03.725436 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:03.725420 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68-kube-api-access-vbcwc" pod "742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68" (UID: "742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68"). InnerVolumeSpecName "kube-api-access-vbcwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:03.824603 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:03.824576 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbcwc\" (UniqueName: \"kubernetes.io/projected/742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68-kube-api-access-vbcwc\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:03.824603 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:03.824602 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68-maas-api-openshift-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:03.824791 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:03.824613 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68-maas-api-xks-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:04.527733 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:04.527702 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z" Nov 03 21:17:04.527958 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:04.527737 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z" event={"ID":"742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68","Type":"ContainerDied","Data":"55c661d2e6c4401e32385380abc7f0dd00f93a4e7fb3dade52eee6ecf2241525"} Nov 03 21:17:04.527958 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:04.527787 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55c661d2e6c4401e32385380abc7f0dd00f93a4e7fb3dade52eee6ecf2241525" Nov 03 21:17:11.903099 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:11.903064 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g"] Nov 03 21:17:12.013667 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:12.013632 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb"] Nov 03 21:17:16.903868 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:16.903807 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g" podUID="87af162e-e529-43f8-9673-0af5fcdc9d7a" containerName="payload-processing" containerID="cri-o://b172430e99560b6ea9cccefc15795ef9c68f249abf21f90cd04959d8dff1d2d3" gracePeriod=25 Nov 03 21:17:17.014429 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.014391 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb" podUID="ea30227a-a99c-4d5f-ba2b-3e6230c29c2e" containerName="payload-pre-processing" containerID="cri-o://f54b102006160ef7a2218573ca363fb5322407ae62db4b3b52a22207aacee706" gracePeriod=25 Nov 03 21:17:17.163271 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.163207 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g" Nov 03 21:17:17.270094 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.270073 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb" Nov 03 21:17:17.271181 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.271165 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87af162e-e529-43f8-9673-0af5fcdc9d7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87af162e-e529-43f8-9673-0af5fcdc9d7a-plugins-config-volume\") pod \"87af162e-e529-43f8-9673-0af5fcdc9d7a\" (UID: \"87af162e-e529-43f8-9673-0af5fcdc9d7a\") " Nov 03 21:17:17.271304 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.271292 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87af162e-e529-43f8-9673-0af5fcdc9d7a-kube-api-access-k46ww\" (UniqueName: \"kubernetes.io/projected/87af162e-e529-43f8-9673-0af5fcdc9d7a-kube-api-access-k46ww\") pod \"87af162e-e529-43f8-9673-0af5fcdc9d7a\" (UID: \"87af162e-e529-43f8-9673-0af5fcdc9d7a\") " Nov 03 21:17:17.271593 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.271568 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87af162e-e529-43f8-9673-0af5fcdc9d7a-plugins-config-volume" pod "87af162e-e529-43f8-9673-0af5fcdc9d7a" (UID: "87af162e-e529-43f8-9673-0af5fcdc9d7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:17.273500 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.273482 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87af162e-e529-43f8-9673-0af5fcdc9d7a-kube-api-access-k46ww" pod "87af162e-e529-43f8-9673-0af5fcdc9d7a" (UID: "87af162e-e529-43f8-9673-0af5fcdc9d7a"). InnerVolumeSpecName "kube-api-access-k46ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:17.372132 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.372087 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea30227a-a99c-4d5f-ba2b-3e6230c29c2e-kube-api-access-c4v2v\" (UniqueName: \"kubernetes.io/projected/ea30227a-a99c-4d5f-ba2b-3e6230c29c2e-kube-api-access-c4v2v\") pod \"ea30227a-a99c-4d5f-ba2b-3e6230c29c2e\" (UID: \"ea30227a-a99c-4d5f-ba2b-3e6230c29c2e\") " Nov 03 21:17:17.372366 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.372262 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea30227a-a99c-4d5f-ba2b-3e6230c29c2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea30227a-a99c-4d5f-ba2b-3e6230c29c2e-plugins-config-volume\") pod \"ea30227a-a99c-4d5f-ba2b-3e6230c29c2e\" (UID: \"ea30227a-a99c-4d5f-ba2b-3e6230c29c2e\") " Nov 03 21:17:17.372615 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.372595 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87af162e-e529-43f8-9673-0af5fcdc9d7a-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.372708 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.372620 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k46ww\" (UniqueName: \"kubernetes.io/projected/87af162e-e529-43f8-9673-0af5fcdc9d7a-kube-api-access-k46ww\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.373318 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.373288 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea30227a-a99c-4d5f-ba2b-3e6230c29c2e-plugins-config-volume" pod "ea30227a-a99c-4d5f-ba2b-3e6230c29c2e" (UID: "ea30227a-a99c-4d5f-ba2b-3e6230c29c2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:17.377540 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.377489 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea30227a-a99c-4d5f-ba2b-3e6230c29c2e-kube-api-access-c4v2v" pod "ea30227a-a99c-4d5f-ba2b-3e6230c29c2e" (UID: "ea30227a-a99c-4d5f-ba2b-3e6230c29c2e"). InnerVolumeSpecName "kube-api-access-c4v2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:17.473730 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.473648 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea30227a-a99c-4d5f-ba2b-3e6230c29c2e-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.473730 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.473675 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4v2v\" (UniqueName: \"kubernetes.io/projected/ea30227a-a99c-4d5f-ba2b-3e6230c29c2e-kube-api-access-c4v2v\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.594513 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.594479 4493 generic.go:356] "Generic (PLEG): container finished" podID="87af162e-e529-43f8-9673-0af5fcdc9d7a" containerID="b172430e99560b6ea9cccefc15795ef9c68f249abf21f90cd04959d8dff1d2d3" exitCode=0 Nov 03 21:17:17.594711 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.594527 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g" Nov 03 21:17:17.594711 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.594557 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g" event={"ID":"87af162e-e529-43f8-9673-0af5fcdc9d7a","Type":"ContainerDied","Data":"b172430e99560b6ea9cccefc15795ef9c68f249abf21f90cd04959d8dff1d2d3"} Nov 03 21:17:17.594711 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.594609 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g" event={"ID":"87af162e-e529-43f8-9673-0af5fcdc9d7a","Type":"ContainerDied","Data":"43325ef54fb37269f62988842507e2c4440df3f2d11041d55685f8ab608fd542"} Nov 03 21:17:17.594711 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.594632 4493 scope.go:122] "RemoveContainer" containerID="b172430e99560b6ea9cccefc15795ef9c68f249abf21f90cd04959d8dff1d2d3" Nov 03 21:17:17.596044 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.596019 4493 generic.go:356] "Generic (PLEG): container finished" podID="ea30227a-a99c-4d5f-ba2b-3e6230c29c2e" containerID="f54b102006160ef7a2218573ca363fb5322407ae62db4b3b52a22207aacee706" exitCode=0 Nov 03 21:17:17.596200 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.596050 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb" event={"ID":"ea30227a-a99c-4d5f-ba2b-3e6230c29c2e","Type":"ContainerDied","Data":"f54b102006160ef7a2218573ca363fb5322407ae62db4b3b52a22207aacee706"} Nov 03 21:17:17.596200 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.596105 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb" event={"ID":"ea30227a-a99c-4d5f-ba2b-3e6230c29c2e","Type":"ContainerDied","Data":"8a83681fae2cb55320e430ad2fe71deb9a265e1673c6a6f06dfb764a621e5953"} Nov 03 21:17:17.596200 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.596060 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb" Nov 03 21:17:17.606869 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.606842 4493 scope.go:122] "RemoveContainer" containerID="b172430e99560b6ea9cccefc15795ef9c68f249abf21f90cd04959d8dff1d2d3" Nov 03 21:17:17.607171 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:17:17.607152 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b172430e99560b6ea9cccefc15795ef9c68f249abf21f90cd04959d8dff1d2d3\": container with ID starting with b172430e99560b6ea9cccefc15795ef9c68f249abf21f90cd04959d8dff1d2d3 not found: ID does not exist" containerID="b172430e99560b6ea9cccefc15795ef9c68f249abf21f90cd04959d8dff1d2d3" Nov 03 21:17:17.607235 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.607178 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b172430e99560b6ea9cccefc15795ef9c68f249abf21f90cd04959d8dff1d2d3"} err="failed to get container status \"b172430e99560b6ea9cccefc15795ef9c68f249abf21f90cd04959d8dff1d2d3\": rpc error: code = NotFound desc = could not find container \"b172430e99560b6ea9cccefc15795ef9c68f249abf21f90cd04959d8dff1d2d3\": container with ID starting with b172430e99560b6ea9cccefc15795ef9c68f249abf21f90cd04959d8dff1d2d3 not found: ID does not exist" Nov 03 21:17:17.607235 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.607195 4493 scope.go:122] "RemoveContainer" containerID="f54b102006160ef7a2218573ca363fb5322407ae62db4b3b52a22207aacee706" Nov 03 21:17:17.616297 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.616279 4493 scope.go:122] "RemoveContainer" containerID="f54b102006160ef7a2218573ca363fb5322407ae62db4b3b52a22207aacee706" Nov 03 21:17:17.616565 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:17:17.616547 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f54b102006160ef7a2218573ca363fb5322407ae62db4b3b52a22207aacee706\": container with ID starting with f54b102006160ef7a2218573ca363fb5322407ae62db4b3b52a22207aacee706 not found: ID does not exist" containerID="f54b102006160ef7a2218573ca363fb5322407ae62db4b3b52a22207aacee706" Nov 03 21:17:17.616634 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.616570 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f54b102006160ef7a2218573ca363fb5322407ae62db4b3b52a22207aacee706"} err="failed to get container status \"f54b102006160ef7a2218573ca363fb5322407ae62db4b3b52a22207aacee706\": rpc error: code = NotFound desc = could not find container \"f54b102006160ef7a2218573ca363fb5322407ae62db4b3b52a22207aacee706\": container with ID starting with f54b102006160ef7a2218573ca363fb5322407ae62db4b3b52a22207aacee706 not found: ID does not exist" Nov 03 21:17:17.766067 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.765979 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb"] Nov 03 21:17:17.771551 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.771527 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e862b-5df6ffd67-dbgtb"] Nov 03 21:17:17.901988 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.901957 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g"] Nov 03 21:17:17.913674 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:17.913646 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1e862b-6cf8f8bbfc-jw47g"] Nov 03 21:17:18.560559 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:18.560522 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87af162e-e529-43f8-9673-0af5fcdc9d7a" path="/var/lib/kubelet/pods/87af162e-e529-43f8-9673-0af5fcdc9d7a/volumes" Nov 03 21:17:18.560994 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:18.560973 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea30227a-a99c-4d5f-ba2b-3e6230c29c2e" path="/var/lib/kubelet/pods/ea30227a-a99c-4d5f-ba2b-3e6230c29c2e/volumes" Nov 03 21:17:21.743840 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:17:21.743806 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:17:25.113288 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:25.113254 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z"] Nov 03 21:17:25.164801 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:25.164774 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e862b-zbb7z"] Nov 03 21:17:26.560714 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:26.560678 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68" path="/var/lib/kubelet/pods/742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68/volumes" Nov 03 21:17:31.625682 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:31.625591 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5"] Nov 03 21:17:31.626177 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:31.625845 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" podUID="c73e8310-a7fd-439b-8f6a-95e99d528e5d" containerName="istio-proxy" containerID="cri-o://c7a20fa64416465ab5b5500c1dd2bb9f439ba6035a3d69d0d8fe48ea51fdb4c5" gracePeriod=30 Nov 03 21:17:36.883286 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:36.883252 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 21:17:37.054802 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.054773 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn"] Nov 03 21:17:37.065602 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.065574 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-data\") pod \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " Nov 03 21:17:37.065762 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.065632 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-token\" (UniqueName: \"kubernetes.io/projected/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-token\") pod \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " Nov 03 21:17:37.065888 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.065681 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-workload-socket\") pod \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " Nov 03 21:17:37.065888 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.065876 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-data" pod "c73e8310-a7fd-439b-8f6a-95e99d528e5d" (UID: "c73e8310-a7fd-439b-8f6a-95e99d528e5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:37.066056 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.065983 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-podinfo\") pod \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " Nov 03 21:17:37.066111 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.066050 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c73e8310-a7fd-439b-8f6a-95e99d528e5d-kube-api-access-nxpjk\" (UniqueName: \"kubernetes.io/projected/c73e8310-a7fd-439b-8f6a-95e99d528e5d-kube-api-access-nxpjk\") pod \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " Nov 03 21:17:37.066111 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.066090 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-workload-certs\") pod \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " Nov 03 21:17:37.066216 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.066123 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istiod-ca-cert\") pod \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " Nov 03 21:17:37.066216 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.066149 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-workload-socket" pod "c73e8310-a7fd-439b-8f6a-95e99d528e5d" (UID: "c73e8310-a7fd-439b-8f6a-95e99d528e5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:37.066216 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.066178 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-envoy\") pod \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " Nov 03 21:17:37.066414 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.066217 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-credential-socket\") pod \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\" (UID: \"c73e8310-a7fd-439b-8f6a-95e99d528e5d\") " Nov 03 21:17:37.066525 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.066503 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istiod-ca-cert" pod "c73e8310-a7fd-439b-8f6a-95e99d528e5d" (UID: "c73e8310-a7fd-439b-8f6a-95e99d528e5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:37.066613 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.066596 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:37.066678 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.066622 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:37.066678 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.066638 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:37.066918 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.066893 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-workload-certs" pod "c73e8310-a7fd-439b-8f6a-95e99d528e5d" (UID: "c73e8310-a7fd-439b-8f6a-95e99d528e5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:37.067016 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.066910 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-credential-socket" pod "c73e8310-a7fd-439b-8f6a-95e99d528e5d" (UID: "c73e8310-a7fd-439b-8f6a-95e99d528e5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:37.068467 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.068437 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-podinfo" pod "c73e8310-a7fd-439b-8f6a-95e99d528e5d" (UID: "c73e8310-a7fd-439b-8f6a-95e99d528e5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:17:37.068975 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.068949 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73e8310-a7fd-439b-8f6a-95e99d528e5d-kube-api-access-nxpjk" pod "c73e8310-a7fd-439b-8f6a-95e99d528e5d" (UID: "c73e8310-a7fd-439b-8f6a-95e99d528e5d"). InnerVolumeSpecName "kube-api-access-nxpjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:37.069079 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.068974 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-envoy" pod "c73e8310-a7fd-439b-8f6a-95e99d528e5d" (UID: "c73e8310-a7fd-439b-8f6a-95e99d528e5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:37.069125 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.069089 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-token" pod "c73e8310-a7fd-439b-8f6a-95e99d528e5d" (UID: "c73e8310-a7fd-439b-8f6a-95e99d528e5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:37.167797 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.167770 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:37.167797 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.167796 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxpjk\" (UniqueName: \"kubernetes.io/projected/c73e8310-a7fd-439b-8f6a-95e99d528e5d-kube-api-access-nxpjk\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:37.167969 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.167806 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:37.167969 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.167815 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:37.167969 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.167823 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73e8310-a7fd-439b-8f6a-95e99d528e5d-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:37.167969 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.167831 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73e8310-a7fd-439b-8f6a-95e99d528e5d-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:37.697225 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.697194 4493 generic.go:356] "Generic (PLEG): container finished" podID="c73e8310-a7fd-439b-8f6a-95e99d528e5d" containerID="c7a20fa64416465ab5b5500c1dd2bb9f439ba6035a3d69d0d8fe48ea51fdb4c5" exitCode=0 Nov 03 21:17:37.697431 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.697237 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" event={"ID":"c73e8310-a7fd-439b-8f6a-95e99d528e5d","Type":"ContainerDied","Data":"c7a20fa64416465ab5b5500c1dd2bb9f439ba6035a3d69d0d8fe48ea51fdb4c5"} Nov 03 21:17:37.697431 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.697253 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" Nov 03 21:17:37.697431 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.697286 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5" event={"ID":"c73e8310-a7fd-439b-8f6a-95e99d528e5d","Type":"ContainerDied","Data":"3c13484ca5f260a742d8892bded4bcab8fd359f0630601e561e4fd3a6523c3f8"} Nov 03 21:17:37.697431 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.697309 4493 scope.go:122] "RemoveContainer" containerID="c7a20fa64416465ab5b5500c1dd2bb9f439ba6035a3d69d0d8fe48ea51fdb4c5" Nov 03 21:17:37.712270 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.712245 4493 scope.go:122] "RemoveContainer" containerID="9164054bb75d12a8e0d3943ad1a0b1c91cc7ac038ac21cfd3fea388053a265b0" Nov 03 21:17:37.721241 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.721218 4493 scope.go:122] "RemoveContainer" containerID="c7a20fa64416465ab5b5500c1dd2bb9f439ba6035a3d69d0d8fe48ea51fdb4c5" Nov 03 21:17:37.721514 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:17:37.721494 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a20fa64416465ab5b5500c1dd2bb9f439ba6035a3d69d0d8fe48ea51fdb4c5\": container with ID starting with c7a20fa64416465ab5b5500c1dd2bb9f439ba6035a3d69d0d8fe48ea51fdb4c5 not found: ID does not exist" containerID="c7a20fa64416465ab5b5500c1dd2bb9f439ba6035a3d69d0d8fe48ea51fdb4c5" Nov 03 21:17:37.721600 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.721526 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a20fa64416465ab5b5500c1dd2bb9f439ba6035a3d69d0d8fe48ea51fdb4c5"} err="failed to get container status \"c7a20fa64416465ab5b5500c1dd2bb9f439ba6035a3d69d0d8fe48ea51fdb4c5\": rpc error: code = NotFound desc = could not find container \"c7a20fa64416465ab5b5500c1dd2bb9f439ba6035a3d69d0d8fe48ea51fdb4c5\": container with ID starting with c7a20fa64416465ab5b5500c1dd2bb9f439ba6035a3d69d0d8fe48ea51fdb4c5 not found: ID does not exist" Nov 03 21:17:37.721600 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.721552 4493 scope.go:122] "RemoveContainer" containerID="9164054bb75d12a8e0d3943ad1a0b1c91cc7ac038ac21cfd3fea388053a265b0" Nov 03 21:17:37.721804 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:17:37.721783 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9164054bb75d12a8e0d3943ad1a0b1c91cc7ac038ac21cfd3fea388053a265b0\": container with ID starting with 9164054bb75d12a8e0d3943ad1a0b1c91cc7ac038ac21cfd3fea388053a265b0 not found: ID does not exist" containerID="9164054bb75d12a8e0d3943ad1a0b1c91cc7ac038ac21cfd3fea388053a265b0" Nov 03 21:17:37.721850 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.721810 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9164054bb75d12a8e0d3943ad1a0b1c91cc7ac038ac21cfd3fea388053a265b0"} err="failed to get container status \"9164054bb75d12a8e0d3943ad1a0b1c91cc7ac038ac21cfd3fea388053a265b0\": rpc error: code = NotFound desc = could not find container \"9164054bb75d12a8e0d3943ad1a0b1c91cc7ac038ac21cfd3fea388053a265b0\": container with ID starting with 9164054bb75d12a8e0d3943ad1a0b1c91cc7ac038ac21cfd3fea388053a265b0 not found: ID does not exist" Nov 03 21:17:37.869127 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.869096 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5"] Nov 03 21:17:37.920524 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:37.920490 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1e862b-openshift-default-5b49d8f879-k6rn5"] Nov 03 21:17:38.560305 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:38.560274 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c73e8310-a7fd-439b-8f6a-95e99d528e5d" path="/var/lib/kubelet/pods/c73e8310-a7fd-439b-8f6a-95e99d528e5d/volumes" Nov 03 21:17:51.744009 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:17:51.743975 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:17:52.075817 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.075754 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" containerID="cri-o://83b7b2649e6d9bdb960537f02bf9f5a6a071cb5bfb9e12b7c8cc5cc621b4deda" gracePeriod=45 Nov 03 21:17:52.331445 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.331382 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 21:17:52.410512 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.410481 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70624c1d-a6b0-4131-97dc-1774fda506ad-kube-api-access-5s74t\" (UniqueName: \"kubernetes.io/projected/70624c1d-a6b0-4131-97dc-1774fda506ad-kube-api-access-5s74t\") pod \"70624c1d-a6b0-4131-97dc-1774fda506ad\" (UID: \"70624c1d-a6b0-4131-97dc-1774fda506ad\") " Nov 03 21:17:52.410698 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.410526 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-dshm\") pod \"70624c1d-a6b0-4131-97dc-1774fda506ad\" (UID: \"70624c1d-a6b0-4131-97dc-1774fda506ad\") " Nov 03 21:17:52.410698 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.410558 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-tmp-dir\") pod \"70624c1d-a6b0-4131-97dc-1774fda506ad\" (UID: \"70624c1d-a6b0-4131-97dc-1774fda506ad\") " Nov 03 21:17:52.410698 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.410586 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-model-cache\") pod \"70624c1d-a6b0-4131-97dc-1774fda506ad\" (UID: \"70624c1d-a6b0-4131-97dc-1774fda506ad\") " Nov 03 21:17:52.410698 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.410687 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-home\" (UniqueName: \"kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-home\") pod \"70624c1d-a6b0-4131-97dc-1774fda506ad\" (UID: \"70624c1d-a6b0-4131-97dc-1774fda506ad\") " Nov 03 21:17:52.410932 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.410743 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70624c1d-a6b0-4131-97dc-1774fda506ad-tls-certs\" (UniqueName: \"kubernetes.io/secret/70624c1d-a6b0-4131-97dc-1774fda506ad-tls-certs\") pod \"70624c1d-a6b0-4131-97dc-1774fda506ad\" (UID: \"70624c1d-a6b0-4131-97dc-1774fda506ad\") " Nov 03 21:17:52.410932 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.410854 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-tmp-dir" pod "70624c1d-a6b0-4131-97dc-1774fda506ad" (UID: "70624c1d-a6b0-4131-97dc-1774fda506ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:52.411015 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.410940 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-model-cache" pod "70624c1d-a6b0-4131-97dc-1774fda506ad" (UID: "70624c1d-a6b0-4131-97dc-1774fda506ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:52.411015 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.410952 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-home" pod "70624c1d-a6b0-4131-97dc-1774fda506ad" (UID: "70624c1d-a6b0-4131-97dc-1774fda506ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:52.411103 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.411066 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-home\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:52.411103 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.411083 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-tmp-dir\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:52.411103 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.411092 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-model-cache\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:52.413073 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.413054 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70624c1d-a6b0-4131-97dc-1774fda506ad-tls-certs" pod "70624c1d-a6b0-4131-97dc-1774fda506ad" (UID: "70624c1d-a6b0-4131-97dc-1774fda506ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:17:52.413208 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.413191 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-dshm" pod "70624c1d-a6b0-4131-97dc-1774fda506ad" (UID: "70624c1d-a6b0-4131-97dc-1774fda506ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:52.413260 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.413204 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70624c1d-a6b0-4131-97dc-1774fda506ad-kube-api-access-5s74t" pod "70624c1d-a6b0-4131-97dc-1774fda506ad" (UID: "70624c1d-a6b0-4131-97dc-1774fda506ad"). InnerVolumeSpecName "kube-api-access-5s74t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:52.511700 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.511663 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70624c1d-a6b0-4131-97dc-1774fda506ad-tls-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:52.511700 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.511700 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s74t\" (UniqueName: \"kubernetes.io/projected/70624c1d-a6b0-4131-97dc-1774fda506ad-kube-api-access-5s74t\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:52.511700 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.511709 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70624c1d-a6b0-4131-97dc-1774fda506ad-dshm\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:52.770036 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.769952 4493 generic.go:356] "Generic (PLEG): container finished" podID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerID="83b7b2649e6d9bdb960537f02bf9f5a6a071cb5bfb9e12b7c8cc5cc621b4deda" exitCode=0 Nov 03 21:17:52.770036 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.769995 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" Nov 03 21:17:52.770553 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.770031 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" event={"ID":"70624c1d-a6b0-4131-97dc-1774fda506ad","Type":"ContainerDied","Data":"83b7b2649e6d9bdb960537f02bf9f5a6a071cb5bfb9e12b7c8cc5cc621b4deda"} Nov 03 21:17:52.770553 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.770079 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn" event={"ID":"70624c1d-a6b0-4131-97dc-1774fda506ad","Type":"ContainerDied","Data":"2f0ba972cce19c348f1182c16156f124f1c35220b890a7efd8df6a17b91d38dc"} Nov 03 21:17:52.770553 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.770100 4493 scope.go:122] "RemoveContainer" containerID="83b7b2649e6d9bdb960537f02bf9f5a6a071cb5bfb9e12b7c8cc5cc621b4deda" Nov 03 21:17:52.778564 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.778545 4493 scope.go:122] "RemoveContainer" containerID="83b7b2649e6d9bdb960537f02bf9f5a6a071cb5bfb9e12b7c8cc5cc621b4deda" Nov 03 21:17:52.778852 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:17:52.778830 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83b7b2649e6d9bdb960537f02bf9f5a6a071cb5bfb9e12b7c8cc5cc621b4deda\": container with ID starting with 83b7b2649e6d9bdb960537f02bf9f5a6a071cb5bfb9e12b7c8cc5cc621b4deda not found: ID does not exist" containerID="83b7b2649e6d9bdb960537f02bf9f5a6a071cb5bfb9e12b7c8cc5cc621b4deda" Nov 03 21:17:52.778899 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.778861 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b7b2649e6d9bdb960537f02bf9f5a6a071cb5bfb9e12b7c8cc5cc621b4deda"} err="failed to get container status \"83b7b2649e6d9bdb960537f02bf9f5a6a071cb5bfb9e12b7c8cc5cc621b4deda\": rpc error: code = NotFound desc = could not find container \"83b7b2649e6d9bdb960537f02bf9f5a6a071cb5bfb9e12b7c8cc5cc621b4deda\": container with ID starting with 83b7b2649e6d9bdb960537f02bf9f5a6a071cb5bfb9e12b7c8cc5cc621b4deda not found: ID does not exist" Nov 03 21:17:52.938305 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.938273 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn"] Nov 03 21:17:52.943659 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:52.943633 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-fe2825/distinct-w3-fe2825-kserve-59d4747868-cldgn"] Nov 03 21:17:54.560562 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:54.560532 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" path="/var/lib/kubelet/pods/70624c1d-a6b0-4131-97dc-1774fda506ad/volumes" Nov 03 21:17:58.765848 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:58.765813 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l"] Nov 03 21:17:58.766316 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:58.766036 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l" podUID="fb5bc208-96eb-49b3-8501-a5f2744fd3e9" containerName="maas-api" containerID="cri-o://4f6904e1aac1d68c7d079e907780a54e289683de4fc8cc4950b289c41f8485af" gracePeriod=30 Nov 03 21:17:59.020204 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:59.020145 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l" Nov 03 21:17:59.070775 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:59.070741 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-openshift-service-ca\") pod \"fb5bc208-96eb-49b3-8501-a5f2744fd3e9\" (UID: \"fb5bc208-96eb-49b3-8501-a5f2744fd3e9\") " Nov 03 21:17:59.070935 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:59.070891 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-maas-api-tls\") pod \"fb5bc208-96eb-49b3-8501-a5f2744fd3e9\" (UID: \"fb5bc208-96eb-49b3-8501-a5f2744fd3e9\") " Nov 03 21:17:59.071020 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:59.070937 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-kube-api-access-vzfkz\" (UniqueName: \"kubernetes.io/projected/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-kube-api-access-vzfkz\") pod \"fb5bc208-96eb-49b3-8501-a5f2744fd3e9\" (UID: \"fb5bc208-96eb-49b3-8501-a5f2744fd3e9\") " Nov 03 21:17:59.071020 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:59.070995 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-metrics-certs\") pod \"fb5bc208-96eb-49b3-8501-a5f2744fd3e9\" (UID: \"fb5bc208-96eb-49b3-8501-a5f2744fd3e9\") " Nov 03 21:17:59.071167 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:59.071135 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-openshift-service-ca" pod "fb5bc208-96eb-49b3-8501-a5f2744fd3e9" (UID: "fb5bc208-96eb-49b3-8501-a5f2744fd3e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:59.071347 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:59.071315 4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-openshift-service-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:59.073226 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:59.073197 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-maas-api-tls" pod "fb5bc208-96eb-49b3-8501-a5f2744fd3e9" (UID: "fb5bc208-96eb-49b3-8501-a5f2744fd3e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:17:59.073350 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:59.073255 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-metrics-certs" pod "fb5bc208-96eb-49b3-8501-a5f2744fd3e9" (UID: "fb5bc208-96eb-49b3-8501-a5f2744fd3e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:17:59.073350 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:59.073284 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-kube-api-access-vzfkz" pod "fb5bc208-96eb-49b3-8501-a5f2744fd3e9" (UID: "fb5bc208-96eb-49b3-8501-a5f2744fd3e9"). InnerVolumeSpecName "kube-api-access-vzfkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:59.172782 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:59.172748 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-maas-api-tls\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:59.172782 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:59.172781 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzfkz\" (UniqueName: \"kubernetes.io/projected/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-kube-api-access-vzfkz\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:59.172974 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:59.172794 4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb5bc208-96eb-49b3-8501-a5f2744fd3e9-metrics-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:59.804574 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:59.804538 4493 generic.go:356] "Generic (PLEG): container finished" podID="fb5bc208-96eb-49b3-8501-a5f2744fd3e9" containerID="4f6904e1aac1d68c7d079e907780a54e289683de4fc8cc4950b289c41f8485af" exitCode=0 Nov 03 21:17:59.804985 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:59.804593 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l" Nov 03 21:17:59.804985 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:59.804620 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l" event={"ID":"fb5bc208-96eb-49b3-8501-a5f2744fd3e9","Type":"ContainerDied","Data":"4f6904e1aac1d68c7d079e907780a54e289683de4fc8cc4950b289c41f8485af"} Nov 03 21:17:59.804985 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:59.804678 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l" event={"ID":"fb5bc208-96eb-49b3-8501-a5f2744fd3e9","Type":"ContainerDied","Data":"1789d1e056f298fbbb6abc9c1adb22795d8d496a09637cf9a791228c81e6359e"} Nov 03 21:17:59.804985 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:59.804699 4493 scope.go:122] "RemoveContainer" containerID="4f6904e1aac1d68c7d079e907780a54e289683de4fc8cc4950b289c41f8485af" Nov 03 21:17:59.814188 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:59.814171 4493 scope.go:122] "RemoveContainer" containerID="4f6904e1aac1d68c7d079e907780a54e289683de4fc8cc4950b289c41f8485af" Nov 03 21:17:59.814469 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:17:59.814454 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f6904e1aac1d68c7d079e907780a54e289683de4fc8cc4950b289c41f8485af\": container with ID starting with 4f6904e1aac1d68c7d079e907780a54e289683de4fc8cc4950b289c41f8485af not found: ID does not exist" containerID="4f6904e1aac1d68c7d079e907780a54e289683de4fc8cc4950b289c41f8485af" Nov 03 21:17:59.814527 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:59.814478 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f6904e1aac1d68c7d079e907780a54e289683de4fc8cc4950b289c41f8485af"} err="failed to get container status \"4f6904e1aac1d68c7d079e907780a54e289683de4fc8cc4950b289c41f8485af\": rpc error: code = NotFound desc = could not find container \"4f6904e1aac1d68c7d079e907780a54e289683de4fc8cc4950b289c41f8485af\": container with ID starting with 4f6904e1aac1d68c7d079e907780a54e289683de4fc8cc4950b289c41f8485af not found: ID does not exist" Nov 03 21:17:59.973189 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:17:59.973155 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l"] Nov 03 21:18:00.025156 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:00.025124 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-fe2825-54bf96c456-p622l"] Nov 03 21:18:00.560655 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:00.560621 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb5bc208-96eb-49b3-8501-a5f2744fd3e9" path="/var/lib/kubelet/pods/fb5bc208-96eb-49b3-8501-a5f2744fd3e9/volumes" Nov 03 21:18:02.761280 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:02.761245 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws"] Nov 03 21:18:02.761878 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:02.761460 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" podUID="00c17f9c-7151-410b-99e4-e09c03c16171" containerName="istio-proxy" containerID="cri-o://c487b03955d5bb80c93c91cda3525c4e8041b9eabaa94fab1ed70332646043ae" gracePeriod=30 Nov 03 21:18:07.394203 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:07.394167 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-mppws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:18:07.394800 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:07.394224 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" podUID="00c17f9c-7151-410b-99e4-e09c03c16171" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:18:08.011391 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.011364 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:18:08.157097 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.157059 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c17f9c-7151-410b-99e4-e09c03c16171-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00c17f9c-7151-410b-99e4-e09c03c16171-istiod-ca-cert\") pod \"00c17f9c-7151-410b-99e4-e09c03c16171\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " Nov 03 21:18:08.157284 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.157129 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-workload-certs\") pod \"00c17f9c-7151-410b-99e4-e09c03c16171\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " Nov 03 21:18:08.157284 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.157225 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-credential-socket\") pod \"00c17f9c-7151-410b-99e4-e09c03c16171\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " Nov 03 21:18:08.157432 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.157311 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-workload-socket\") pod \"00c17f9c-7151-410b-99e4-e09c03c16171\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " Nov 03 21:18:08.157432 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.157362 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c17f9c-7151-410b-99e4-e09c03c16171-istio-token\" (UniqueName: \"kubernetes.io/projected/00c17f9c-7151-410b-99e4-e09c03c16171-istio-token\") pod \"00c17f9c-7151-410b-99e4-e09c03c16171\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " Nov 03 21:18:08.157432 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.157402 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-istio-envoy\") pod \"00c17f9c-7151-410b-99e4-e09c03c16171\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " Nov 03 21:18:08.157599 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.157460 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-workload-certs" pod "00c17f9c-7151-410b-99e4-e09c03c16171" (UID: "00c17f9c-7151-410b-99e4-e09c03c16171"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:08.157599 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.157464 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c17f9c-7151-410b-99e4-e09c03c16171-kube-api-access-69dpz\" (UniqueName: \"kubernetes.io/projected/00c17f9c-7151-410b-99e4-e09c03c16171-kube-api-access-69dpz\") pod \"00c17f9c-7151-410b-99e4-e09c03c16171\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " Nov 03 21:18:08.157599 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.157481 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c17f9c-7151-410b-99e4-e09c03c16171-istiod-ca-cert" pod "00c17f9c-7151-410b-99e4-e09c03c16171" (UID: "00c17f9c-7151-410b-99e4-e09c03c16171"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:18:08.157757 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.157604 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-istio-data\") pod \"00c17f9c-7151-410b-99e4-e09c03c16171\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " Nov 03 21:18:08.157757 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.157637 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00c17f9c-7151-410b-99e4-e09c03c16171-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00c17f9c-7151-410b-99e4-e09c03c16171-istio-podinfo\") pod \"00c17f9c-7151-410b-99e4-e09c03c16171\" (UID: \"00c17f9c-7151-410b-99e4-e09c03c16171\") " Nov 03 21:18:08.157757 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.157583 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-credential-socket" pod "00c17f9c-7151-410b-99e4-e09c03c16171" (UID: "00c17f9c-7151-410b-99e4-e09c03c16171"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:08.157757 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.157659 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-workload-socket" pod "00c17f9c-7151-410b-99e4-e09c03c16171" (UID: "00c17f9c-7151-410b-99e4-e09c03c16171"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:08.157960 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.157772 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-istio-data" pod "00c17f9c-7151-410b-99e4-e09c03c16171" (UID: "00c17f9c-7151-410b-99e4-e09c03c16171"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:08.158032 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.158011 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.158119 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.158039 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00c17f9c-7151-410b-99e4-e09c03c16171-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.158119 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.158056 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.158119 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.158078 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.158119 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.158098 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.159898 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.159869 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-istio-envoy" pod "00c17f9c-7151-410b-99e4-e09c03c16171" (UID: "00c17f9c-7151-410b-99e4-e09c03c16171"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:08.160027 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.159956 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c17f9c-7151-410b-99e4-e09c03c16171-istio-token" pod "00c17f9c-7151-410b-99e4-e09c03c16171" (UID: "00c17f9c-7151-410b-99e4-e09c03c16171"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:08.160027 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.159987 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c17f9c-7151-410b-99e4-e09c03c16171-kube-api-access-69dpz" pod "00c17f9c-7151-410b-99e4-e09c03c16171" (UID: "00c17f9c-7151-410b-99e4-e09c03c16171"). InnerVolumeSpecName "kube-api-access-69dpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:08.160273 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.160251 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00c17f9c-7151-410b-99e4-e09c03c16171-istio-podinfo" pod "00c17f9c-7151-410b-99e4-e09c03c16171" (UID: "00c17f9c-7151-410b-99e4-e09c03c16171"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:18:08.259013 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.258984 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00c17f9c-7151-410b-99e4-e09c03c16171-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.259013 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.259010 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69dpz\" (UniqueName: \"kubernetes.io/projected/00c17f9c-7151-410b-99e4-e09c03c16171-kube-api-access-69dpz\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.259013 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.259020 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00c17f9c-7151-410b-99e4-e09c03c16171-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.259236 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.259028 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c17f9c-7151-410b-99e4-e09c03c16171-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.851490 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.851395 4493 generic.go:356] "Generic (PLEG): container finished" podID="00c17f9c-7151-410b-99e4-e09c03c16171" containerID="c487b03955d5bb80c93c91cda3525c4e8041b9eabaa94fab1ed70332646043ae" exitCode=0 Nov 03 21:18:08.851490 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.851448 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" Nov 03 21:18:08.851974 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.851487 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" event={"ID":"00c17f9c-7151-410b-99e4-e09c03c16171","Type":"ContainerDied","Data":"c487b03955d5bb80c93c91cda3525c4e8041b9eabaa94fab1ed70332646043ae"} Nov 03 21:18:08.851974 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.851543 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws" event={"ID":"00c17f9c-7151-410b-99e4-e09c03c16171","Type":"ContainerDied","Data":"89afaae80551a2144e281472a885d68e5ee7d2a474d2f0f6f819b03232394485"} Nov 03 21:18:08.851974 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.851566 4493 scope.go:122] "RemoveContainer" containerID="c487b03955d5bb80c93c91cda3525c4e8041b9eabaa94fab1ed70332646043ae" Nov 03 21:18:08.861048 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.861028 4493 scope.go:122] "RemoveContainer" containerID="c487b03955d5bb80c93c91cda3525c4e8041b9eabaa94fab1ed70332646043ae" Nov 03 21:18:08.861373 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:18:08.861352 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c487b03955d5bb80c93c91cda3525c4e8041b9eabaa94fab1ed70332646043ae\": container with ID starting with c487b03955d5bb80c93c91cda3525c4e8041b9eabaa94fab1ed70332646043ae not found: ID does not exist" containerID="c487b03955d5bb80c93c91cda3525c4e8041b9eabaa94fab1ed70332646043ae" Nov 03 21:18:08.861463 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:08.861389 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c487b03955d5bb80c93c91cda3525c4e8041b9eabaa94fab1ed70332646043ae"} err="failed to get container status \"c487b03955d5bb80c93c91cda3525c4e8041b9eabaa94fab1ed70332646043ae\": rpc error: code = NotFound desc = could not find container \"c487b03955d5bb80c93c91cda3525c4e8041b9eabaa94fab1ed70332646043ae\": container with ID starting with c487b03955d5bb80c93c91cda3525c4e8041b9eabaa94fab1ed70332646043ae not found: ID does not exist" Nov 03 21:18:09.073260 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:09.073221 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws"] Nov 03 21:18:09.124865 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:09.124787 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mppws"] Nov 03 21:18:10.560019 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:10.559985 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00c17f9c-7151-410b-99e4-e09c03c16171" path="/var/lib/kubelet/pods/00c17f9c-7151-410b-99e4-e09c03c16171/volumes" Nov 03 21:18:21.743347 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:18:21.743293 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:18:48.127358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127313 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6"] Nov 03 21:18:48.127818 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127685 4493 cpu_manager.go:414] "containerMap: removing container" podUID="87af162e-e529-43f8-9673-0af5fcdc9d7a" containerName="payload-processing" Nov 03 21:18:48.127818 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127697 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87af162e-e529-43f8-9673-0af5fcdc9d7a" containerName="payload-processing" Nov 03 21:18:48.127818 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127714 4493 cpu_manager.go:414] "containerMap: removing container" podUID="fb5bc208-96eb-49b3-8501-a5f2744fd3e9" containerName="maas-api" Nov 03 21:18:48.127818 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127719 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb5bc208-96eb-49b3-8501-a5f2744fd3e9" containerName="maas-api" Nov 03 21:18:48.127818 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127727 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ea30227a-a99c-4d5f-ba2b-3e6230c29c2e" containerName="payload-pre-processing" Nov 03 21:18:48.127818 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127731 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea30227a-a99c-4d5f-ba2b-3e6230c29c2e" containerName="payload-pre-processing" Nov 03 21:18:48.127818 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127740 4493 cpu_manager.go:414] "containerMap: removing container" podUID="00c17f9c-7151-410b-99e4-e09c03c16171" containerName="istio-proxy" Nov 03 21:18:48.127818 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127744 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c17f9c-7151-410b-99e4-e09c03c16171" containerName="istio-proxy" Nov 03 21:18:48.127818 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127753 4493 cpu_manager.go:414] "containerMap: removing container" podUID="c73e8310-a7fd-439b-8f6a-95e99d528e5d" containerName="istio-proxy" Nov 03 21:18:48.127818 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127757 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c73e8310-a7fd-439b-8f6a-95e99d528e5d" containerName="istio-proxy" Nov 03 21:18:48.127818 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127766 4493 cpu_manager.go:414] "containerMap: removing container" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" Nov 03 21:18:48.127818 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127771 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" Nov 03 21:18:48.127818 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127776 4493 cpu_manager.go:414] "containerMap: removing container" podUID="b5b5e9e0-94df-40e7-b031-15c11babc655" containerName="istio-proxy" Nov 03 21:18:48.127818 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127781 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5b5e9e0-94df-40e7-b031-15c11babc655" containerName="istio-proxy" Nov 03 21:18:48.127818 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127792 4493 cpu_manager.go:414] "containerMap: removing container" podUID="c73e8310-a7fd-439b-8f6a-95e99d528e5d" containerName="istio-proxy" Nov 03 21:18:48.127818 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127797 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c73e8310-a7fd-439b-8f6a-95e99d528e5d" containerName="istio-proxy" Nov 03 21:18:48.127818 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127806 4493 cpu_manager.go:414] "containerMap: removing container" podUID="742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68" containerName="revoke-keys" Nov 03 21:18:48.127818 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127810 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68" containerName="revoke-keys" Nov 03 21:18:48.128395 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127869 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="87af162e-e529-43f8-9673-0af5fcdc9d7a" containerName="payload-processing" Nov 03 21:18:48.128395 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127880 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb5bc208-96eb-49b3-8501-a5f2744fd3e9" containerName="maas-api" Nov 03 21:18:48.128395 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127886 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea30227a-a99c-4d5f-ba2b-3e6230c29c2e" containerName="payload-pre-processing" Nov 03 21:18:48.128395 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127892 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c17f9c-7151-410b-99e4-e09c03c16171" containerName="istio-proxy" Nov 03 21:18:48.128395 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127898 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c73e8310-a7fd-439b-8f6a-95e99d528e5d" containerName="istio-proxy" Nov 03 21:18:48.128395 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127903 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="70624c1d-a6b0-4131-97dc-1774fda506ad" containerName="main" Nov 03 21:18:48.128395 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127909 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5b5e9e0-94df-40e7-b031-15c11babc655" containerName="istio-proxy" Nov 03 21:18:48.128395 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.127914 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="742e26fe-9b4b-41a2-bb7b-3a62ff2b5e68" containerName="revoke-keys" Nov 03 21:18:48.130934 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.130918 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.233220 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.233182 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3601ef6-138a-4d85-8bd1-3894244e6496-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.233392 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.233224 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3601ef6-138a-4d85-8bd1-3894244e6496-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.233392 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.233290 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3601ef6-138a-4d85-8bd1-3894244e6496-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.233392 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.233385 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plz78\" (UniqueName: \"kubernetes.io/projected/b3601ef6-138a-4d85-8bd1-3894244e6496-kube-api-access-plz78\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.233513 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.233413 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3601ef6-138a-4d85-8bd1-3894244e6496-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.233513 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.233473 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3601ef6-138a-4d85-8bd1-3894244e6496-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.233513 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.233508 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3601ef6-138a-4d85-8bd1-3894244e6496-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.233616 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.233533 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3601ef6-138a-4d85-8bd1-3894244e6496-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.233616 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.233556 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3601ef6-138a-4d85-8bd1-3894244e6496-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.332735 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.332685 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6"] Nov 03 21:18:48.335027 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.335000 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3601ef6-138a-4d85-8bd1-3894244e6496-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.335169 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.335037 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3601ef6-138a-4d85-8bd1-3894244e6496-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.335169 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.335079 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3601ef6-138a-4d85-8bd1-3894244e6496-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.335169 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.335099 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3601ef6-138a-4d85-8bd1-3894244e6496-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.335169 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.335121 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3601ef6-138a-4d85-8bd1-3894244e6496-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.335169 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.335155 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plz78\" (UniqueName: \"kubernetes.io/projected/b3601ef6-138a-4d85-8bd1-3894244e6496-kube-api-access-plz78\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.335479 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.335175 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3601ef6-138a-4d85-8bd1-3894244e6496-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.335479 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.335204 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3601ef6-138a-4d85-8bd1-3894244e6496-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.335479 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.335250 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3601ef6-138a-4d85-8bd1-3894244e6496-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.335633 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.335525 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3601ef6-138a-4d85-8bd1-3894244e6496-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.335681 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.335657 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3601ef6-138a-4d85-8bd1-3894244e6496-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.335721 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.335701 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3601ef6-138a-4d85-8bd1-3894244e6496-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.335721 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.335713 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3601ef6-138a-4d85-8bd1-3894244e6496-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.336469 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.336444 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3601ef6-138a-4d85-8bd1-3894244e6496-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.337599 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.337578 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3601ef6-138a-4d85-8bd1-3894244e6496-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.338055 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.338030 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3601ef6-138a-4d85-8bd1-3894244e6496-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.393017 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.392957 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plz78\" (UniqueName: \"kubernetes.io/projected/b3601ef6-138a-4d85-8bd1-3894244e6496-kube-api-access-plz78\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.393017 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.392958 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3601ef6-138a-4d85-8bd1-3894244e6496-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-65lk6\" (UID: \"b3601ef6-138a-4d85-8bd1-3894244e6496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.444009 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.443981 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:48.871090 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:18:48.871056 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3601ef6_138a_4d85_8bd1_3894244e6496.slice/crio-d55a4f0fdc16c1610e6bc147d94da5c057a789d295626e72c617628c54a17258 WatchSource:0}: Error finding container d55a4f0fdc16c1610e6bc147d94da5c057a789d295626e72c617628c54a17258: Status 404 returned error can't find the container with id d55a4f0fdc16c1610e6bc147d94da5c057a789d295626e72c617628c54a17258 Nov 03 21:18:48.871238 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.871087 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6"] Nov 03 21:18:48.873448 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.873418 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:18:48.873568 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.873491 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:18:48.873568 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:48.873533 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:18:49.045757 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:49.045721 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" event={"ID":"b3601ef6-138a-4d85-8bd1-3894244e6496","Type":"ContainerStarted","Data":"6dde128f7e78e181d2ccdd34cbb8b7e47d65fa3b405896a7a9413bf0cf122bff"} Nov 03 21:18:49.045930 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:49.045785 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" event={"ID":"b3601ef6-138a-4d85-8bd1-3894244e6496","Type":"ContainerStarted","Data":"d55a4f0fdc16c1610e6bc147d94da5c057a789d295626e72c617628c54a17258"} Nov 03 21:18:49.105067 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:49.105040 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 21:18:49.114633 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:49.114612 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 21:18:49.444732 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:49.444700 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:49.446313 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:49.446285 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-65lk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 03 21:18:49.446500 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:49.446359 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" podUID="b3601ef6-138a-4d85-8bd1-3894244e6496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 03 21:18:50.444936 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:50.444901 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-65lk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 03 21:18:50.445361 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:50.444974 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" podUID="b3601ef6-138a-4d85-8bd1-3894244e6496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 03 21:18:51.445043 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:51.445006 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-65lk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 03 21:18:51.445466 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:51.445064 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" podUID="b3601ef6-138a-4d85-8bd1-3894244e6496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 03 21:18:51.744064 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:18:51.743947 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:18:52.445220 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:52.445189 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-65lk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 03 21:18:52.445693 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:52.445668 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" podUID="b3601ef6-138a-4d85-8bd1-3894244e6496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 03 21:18:53.444594 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:53.444551 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-65lk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 03 21:18:53.444979 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:53.444615 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" podUID="b3601ef6-138a-4d85-8bd1-3894244e6496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 03 21:18:54.445033 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:54.444998 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-65lk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 03 21:18:54.445477 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:54.445053 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" podUID="b3601ef6-138a-4d85-8bd1-3894244e6496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 03 21:18:55.445126 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:55.445091 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-65lk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 03 21:18:55.445558 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:55.445143 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" podUID="b3601ef6-138a-4d85-8bd1-3894244e6496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 03 21:18:57.104046 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:57.104010 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:57.104553 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:57.104369 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:57.105418 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:57.105399 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" Nov 03 21:18:57.234857 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:18:57.234800 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" podStartSLOduration=9.234787266 podStartE2EDuration="9.234787266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:18:49.16525904 +0000 UTC m=+3000.723249561" watchObservedRunningTime="2026-11-03 21:18:57.234787266 +0000 UTC m=+3008.792777765" Nov 03 21:19:21.743100 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:19:21.743059 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:19:51.744443 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:19:51.744407 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:20:03.672764 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.672730 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw"] Nov 03 21:20:03.673201 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.673187 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c73e8310-a7fd-439b-8f6a-95e99d528e5d" containerName="istio-proxy" Nov 03 21:20:03.676008 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.675991 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.783119 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.783085 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecce3865-c792-4c76-ba4b-58c563321e8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.783119 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.783129 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecce3865-c792-4c76-ba4b-58c563321e8a-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.783413 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.783145 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecce3865-c792-4c76-ba4b-58c563321e8a-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.783413 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.783285 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecce3865-c792-4c76-ba4b-58c563321e8a-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.783413 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.783364 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecce3865-c792-4c76-ba4b-58c563321e8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.783413 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.783389 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecce3865-c792-4c76-ba4b-58c563321e8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.783413 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.783412 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecce3865-c792-4c76-ba4b-58c563321e8a-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.783699 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.783485 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks2kv\" (UniqueName: \"kubernetes.io/projected/ecce3865-c792-4c76-ba4b-58c563321e8a-kube-api-access-ks2kv\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.783699 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.783518 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecce3865-c792-4c76-ba4b-58c563321e8a-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.783949 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.783928 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw"] Nov 03 21:20:03.884681 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.884652 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecce3865-c792-4c76-ba4b-58c563321e8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.884851 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.884688 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecce3865-c792-4c76-ba4b-58c563321e8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.884851 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.884707 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecce3865-c792-4c76-ba4b-58c563321e8a-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.884851 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.884782 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks2kv\" (UniqueName: \"kubernetes.io/projected/ecce3865-c792-4c76-ba4b-58c563321e8a-kube-api-access-ks2kv\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.884851 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.884820 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecce3865-c792-4c76-ba4b-58c563321e8a-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.885083 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.884899 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecce3865-c792-4c76-ba4b-58c563321e8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.885083 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.884939 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecce3865-c792-4c76-ba4b-58c563321e8a-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.885083 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.884971 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecce3865-c792-4c76-ba4b-58c563321e8a-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.885083 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.885018 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecce3865-c792-4c76-ba4b-58c563321e8a-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.885374 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.885317 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecce3865-c792-4c76-ba4b-58c563321e8a-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.885463 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.885371 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecce3865-c792-4c76-ba4b-58c563321e8a-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.885747 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.885728 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecce3865-c792-4c76-ba4b-58c563321e8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.885747 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.885737 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecce3865-c792-4c76-ba4b-58c563321e8a-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.885850 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.885807 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecce3865-c792-4c76-ba4b-58c563321e8a-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.887240 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.887222 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecce3865-c792-4c76-ba4b-58c563321e8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.887468 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.887451 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecce3865-c792-4c76-ba4b-58c563321e8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.943542 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.943479 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks2kv\" (UniqueName: \"kubernetes.io/projected/ecce3865-c792-4c76-ba4b-58c563321e8a-kube-api-access-ks2kv\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.943542 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.943502 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecce3865-c792-4c76-ba4b-58c563321e8a-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-cqjpw\" (UID: \"ecce3865-c792-4c76-ba4b-58c563321e8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:03.989311 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:03.989289 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:04.413719 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:04.413682 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw"] Nov 03 21:20:04.414186 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:20:04.414160 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecce3865_c792_4c76_ba4b_58c563321e8a.slice/crio-6a747cd5f355cad16e3ed5258ac673c9e22ca73f1a77e6ad4d4e8349b6a72602 WatchSource:0}: Error finding container 6a747cd5f355cad16e3ed5258ac673c9e22ca73f1a77e6ad4d4e8349b6a72602: Status 404 returned error can't find the container with id 6a747cd5f355cad16e3ed5258ac673c9e22ca73f1a77e6ad4d4e8349b6a72602 Nov 03 21:20:04.416326 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:04.416296 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:20:04.416414 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:04.416377 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:20:04.416414 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:04.416408 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:20:05.418856 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:05.418818 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" event={"ID":"ecce3865-c792-4c76-ba4b-58c563321e8a","Type":"ContainerStarted","Data":"1b62e47f836828ad39ac10b49007332ae9dd1bef15824c458b7083aabb0607f1"} Nov 03 21:20:05.419292 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:05.418903 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" event={"ID":"ecce3865-c792-4c76-ba4b-58c563321e8a","Type":"ContainerStarted","Data":"6a747cd5f355cad16e3ed5258ac673c9e22ca73f1a77e6ad4d4e8349b6a72602"} Nov 03 21:20:05.609911 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:05.609846 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" podStartSLOduration=2.609825689 podStartE2EDuration="2.609825689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:20:05.608850845 +0000 UTC m=+3077.166841346" watchObservedRunningTime="2026-11-03 21:20:05.609825689 +0000 UTC m=+3077.167816191" Nov 03 21:20:06.002359 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:05.992965 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:06.002359 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:05.993091 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-cqjpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 03 21:20:06.002359 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:05.993137 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" podUID="ecce3865-c792-4c76-ba4b-58c563321e8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 03 21:20:07.003359 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:06.998754 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-cqjpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 03 21:20:07.003359 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:06.998815 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" podUID="ecce3865-c792-4c76-ba4b-58c563321e8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 03 21:20:08.008481 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:08.008444 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-cqjpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 03 21:20:08.008871 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:08.008513 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" podUID="ecce3865-c792-4c76-ba4b-58c563321e8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 03 21:20:09.003359 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:09.000761 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-cqjpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 03 21:20:09.003359 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:09.000823 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" podUID="ecce3865-c792-4c76-ba4b-58c563321e8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 03 21:20:09.994366 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:09.989774 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-cqjpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 03 21:20:09.994366 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:09.989839 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" podUID="ecce3865-c792-4c76-ba4b-58c563321e8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 03 21:20:10.989755 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:10.989719 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-cqjpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 03 21:20:10.989944 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:10.989786 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" podUID="ecce3865-c792-4c76-ba4b-58c563321e8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 03 21:20:12.592837 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:12.592804 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:12.593403 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:12.593199 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:12.594229 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:20:12.594209 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" Nov 03 21:20:21.744308 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:20:21.744271 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:20:51.744148 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:20:51.744102 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:21:21.745113 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:21:21.745078 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:21:51.744177 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:21:51.744142 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:22:21.744704 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:22:21.744668 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:22:35.235360 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:22:35.235023 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-zgpq2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:22:35.235360 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:22:35.235165 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" podUID="da127cda-8ef4-43c8-8eb8-2fe89b52ffcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:22:35.281521 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:22:35.281471 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-xd49n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:22:35.281896 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:22:35.281859 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" podUID="197b41b5-4233-40c8-855a-38b91aedf4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:22:51.743445 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:22:51.743406 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:23:05.231481 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:23:05.231437 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-zgpq2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:23:05.232015 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:23:05.231494 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" podUID="da127cda-8ef4-43c8-8eb8-2fe89b52ffcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:23:05.255984 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:23:05.255951 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-xd49n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:23:05.256159 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:23:05.256006 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" podUID="197b41b5-4233-40c8-855a-38b91aedf4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:23:21.744036 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:23:21.744001 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:23:49.192817 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:23:49.192785 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 21:23:49.278522 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:23:49.278493 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 21:23:50.490220 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:23:50.490187 4493 scope.go:122] "RemoveContainer" containerID="605752abdbfe18c58f9674c970f85ded053946c95cf9de8a6d5e076d6e3d52f3" Nov 03 21:23:51.744460 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:23:51.744421 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:24:21.743788 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:24:21.743750 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:24:51.744275 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:24:51.744234 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:25:21.804280 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:25:21.804169 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:25:40.156829 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.156793 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf"] Nov 03 21:25:40.160345 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.160317 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.222209 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.222179 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-d9f39a-openshift-default-dockercfg-tnrft\"" Nov 03 21:25:40.269453 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.269422 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf"] Nov 03 21:25:40.275758 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.275729 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-credential-socket\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.275905 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.275767 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73ca984a-209a-431a-81bd-b04175901685-istio-token\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.275905 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.275790 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-workload-certs\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.275905 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.275871 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-istio-envoy\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.276077 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.275933 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73ca984a-209a-431a-81bd-b04175901685-istio-podinfo\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.276077 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.276016 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-workload-socket\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.276245 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.276087 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73ca984a-209a-431a-81bd-b04175901685-istiod-ca-cert\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.276245 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.276155 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9ctf\" (UniqueName: \"kubernetes.io/projected/73ca984a-209a-431a-81bd-b04175901685-kube-api-access-z9ctf\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.276245 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.276207 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-istio-data\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.377153 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.377122 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73ca984a-209a-431a-81bd-b04175901685-istiod-ca-cert\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.377376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.377157 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9ctf\" (UniqueName: \"kubernetes.io/projected/73ca984a-209a-431a-81bd-b04175901685-kube-api-access-z9ctf\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.377376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.377184 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-istio-data\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.377376 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.377222 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-credential-socket\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.377550 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.377375 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73ca984a-209a-431a-81bd-b04175901685-istio-token\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.377550 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.377417 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-workload-certs\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.377550 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.377453 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-istio-envoy\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.377550 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.377496 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73ca984a-209a-431a-81bd-b04175901685-istio-podinfo\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.377768 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.377573 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-workload-socket\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.377768 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.377643 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-credential-socket\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.377768 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.377753 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-workload-certs\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.377971 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.377948 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-workload-socket\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.378031 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.377993 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-istio-data\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.378031 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.378018 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73ca984a-209a-431a-81bd-b04175901685-istiod-ca-cert\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.379844 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.379824 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-istio-envoy\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.380069 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.380042 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73ca984a-209a-431a-81bd-b04175901685-istio-podinfo\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.435811 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.435737 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73ca984a-209a-431a-81bd-b04175901685-istio-token\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.435943 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.435806 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9ctf\" (UniqueName: \"kubernetes.io/projected/73ca984a-209a-431a-81bd-b04175901685-kube-api-access-z9ctf\") pod \"e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.473131 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.473102 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:40.901092 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.901058 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf"] Nov 03 21:25:40.901439 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:25:40.901417 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ca984a_209a_431a_81bd_b04175901685.slice/crio-70ca31905a8a04c792e221ff07e3b306ac9b72831d89b7e21b5ed73775db5cc5 WatchSource:0}: Error finding container 70ca31905a8a04c792e221ff07e3b306ac9b72831d89b7e21b5ed73775db5cc5: Status 404 returned error can't find the container with id 70ca31905a8a04c792e221ff07e3b306ac9b72831d89b7e21b5ed73775db5cc5 Nov 03 21:25:40.903600 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.903563 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:25:40.903677 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.903635 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:25:40.903677 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:40.903664 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:25:41.439849 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:41.439813 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" event={"ID":"73ca984a-209a-431a-81bd-b04175901685","Type":"ContainerStarted","Data":"ad7914966cd99549772a54a490e79104cb02166093f47618c4a4982639a167d0"} Nov 03 21:25:41.440223 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:41.439883 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" event={"ID":"73ca984a-209a-431a-81bd-b04175901685","Type":"ContainerStarted","Data":"70ca31905a8a04c792e221ff07e3b306ac9b72831d89b7e21b5ed73775db5cc5"} Nov 03 21:25:41.474130 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:41.474101 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:41.479065 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:41.479046 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:41.560614 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:41.560568 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" podStartSLOduration=1.560554082 podStartE2EDuration="1.560554082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:41.559180334 +0000 UTC m=+3413.117170834" watchObservedRunningTime="2026-11-03 21:25:41.560554082 +0000 UTC m=+3413.118544581" Nov 03 21:25:42.444583 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:42.444551 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:42.445455 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:42.445438 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:25:47.908215 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:47.908176 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t"] Nov 03 21:25:47.912139 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:47.912122 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t" Nov 03 21:25:47.949944 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:47.949915 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38bde3b-9a8d-4907-aa35-29287d53353f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t\" (UID: \"b38bde3b-9a8d-4907-aa35-29287d53353f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t" Nov 03 21:25:47.950074 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:47.949956 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg6zk\" (UniqueName: \"kubernetes.io/projected/b38bde3b-9a8d-4907-aa35-29287d53353f-kube-api-access-tg6zk\") pod \"payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t\" (UID: \"b38bde3b-9a8d-4907-aa35-29287d53353f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t" Nov 03 21:25:47.964653 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:47.964625 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t"] Nov 03 21:25:48.013995 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:48.013955 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-d9f39a\"" Nov 03 21:25:48.017703 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:48.017665 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-33b1c60b-dockercfg-knjhp\"" Nov 03 21:25:48.050573 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:48.050541 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg6zk\" (UniqueName: \"kubernetes.io/projected/b38bde3b-9a8d-4907-aa35-29287d53353f-kube-api-access-tg6zk\") pod \"payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t\" (UID: \"b38bde3b-9a8d-4907-aa35-29287d53353f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t" Nov 03 21:25:48.050749 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:48.050639 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38bde3b-9a8d-4907-aa35-29287d53353f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t\" (UID: \"b38bde3b-9a8d-4907-aa35-29287d53353f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t" Nov 03 21:25:48.051310 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:48.051283 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38bde3b-9a8d-4907-aa35-29287d53353f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t\" (UID: \"b38bde3b-9a8d-4907-aa35-29287d53353f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t" Nov 03 21:25:48.067867 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:48.067840 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8"] Nov 03 21:25:48.071710 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:48.071693 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8" Nov 03 21:25:48.109062 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:48.109038 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg6zk\" (UniqueName: \"kubernetes.io/projected/b38bde3b-9a8d-4907-aa35-29287d53353f-kube-api-access-tg6zk\") pod \"payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t\" (UID: \"b38bde3b-9a8d-4907-aa35-29287d53353f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t" Nov 03 21:25:48.119095 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:48.119072 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8"] Nov 03 21:25:48.151479 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:48.151451 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8b35668-9b5e-4b65-ac26-66f7311ffb01-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8\" (UID: \"e8b35668-9b5e-4b65-ac26-66f7311ffb01\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8" Nov 03 21:25:48.151623 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:48.151501 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqttb\" (UniqueName: \"kubernetes.io/projected/e8b35668-9b5e-4b65-ac26-66f7311ffb01-kube-api-access-jqttb\") pod \"payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8\" (UID: \"e8b35668-9b5e-4b65-ac26-66f7311ffb01\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8" Nov 03 21:25:48.225698 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:48.225623 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t" Nov 03 21:25:48.252613 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:48.252583 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8b35668-9b5e-4b65-ac26-66f7311ffb01-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8\" (UID: \"e8b35668-9b5e-4b65-ac26-66f7311ffb01\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8" Nov 03 21:25:48.252756 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:48.252641 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqttb\" (UniqueName: \"kubernetes.io/projected/e8b35668-9b5e-4b65-ac26-66f7311ffb01-kube-api-access-jqttb\") pod \"payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8\" (UID: \"e8b35668-9b5e-4b65-ac26-66f7311ffb01\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8" Nov 03 21:25:48.253347 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:48.253309 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8b35668-9b5e-4b65-ac26-66f7311ffb01-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8\" (UID: \"e8b35668-9b5e-4b65-ac26-66f7311ffb01\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8" Nov 03 21:25:48.311414 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:48.311383 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqttb\" (UniqueName: \"kubernetes.io/projected/e8b35668-9b5e-4b65-ac26-66f7311ffb01-kube-api-access-jqttb\") pod \"payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8\" (UID: \"e8b35668-9b5e-4b65-ac26-66f7311ffb01\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8" Nov 03 21:25:48.384673 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:48.384641 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8" Nov 03 21:25:48.597141 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:48.597113 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t"] Nov 03 21:25:48.598973 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:25:48.598942 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38bde3b_9a8d_4907_aa35_29287d53353f.slice/crio-9929b9cc016fd79e39a6defc0c5d0e90a770c4f1bad00af86dda187508dfc79c WatchSource:0}: Error finding container 9929b9cc016fd79e39a6defc0c5d0e90a770c4f1bad00af86dda187508dfc79c: Status 404 returned error can't find the container with id 9929b9cc016fd79e39a6defc0c5d0e90a770c4f1bad00af86dda187508dfc79c Nov 03 21:25:48.759837 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:48.759803 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8"] Nov 03 21:25:48.766365 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:25:48.766321 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b35668_9b5e_4b65_ac26_66f7311ffb01.slice/crio-53aea6bd30c4c9d7d4cd6fa636431ffaa084e523a0cc20b14f2e2e4ff87d152d WatchSource:0}: Error finding container 53aea6bd30c4c9d7d4cd6fa636431ffaa084e523a0cc20b14f2e2e4ff87d152d: Status 404 returned error can't find the container with id 53aea6bd30c4c9d7d4cd6fa636431ffaa084e523a0cc20b14f2e2e4ff87d152d Nov 03 21:25:49.480991 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:49.480949 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t" event={"ID":"b38bde3b-9a8d-4907-aa35-29287d53353f","Type":"ContainerStarted","Data":"726cdeed5e0ea427a93305bfbded79a8bb7575e5d96bf0ce6f3f55399f8331d8"} Nov 03 21:25:49.481501 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:49.481129 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t" Nov 03 21:25:49.481501 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:49.481167 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t" event={"ID":"b38bde3b-9a8d-4907-aa35-29287d53353f","Type":"ContainerStarted","Data":"9929b9cc016fd79e39a6defc0c5d0e90a770c4f1bad00af86dda187508dfc79c"} Nov 03 21:25:49.482580 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:49.482557 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8" event={"ID":"e8b35668-9b5e-4b65-ac26-66f7311ffb01","Type":"ContainerStarted","Data":"e9ff42ff5f4e8c100d72e89092473bd5c7c5a533c5715cf6f4b3a41cd26dc1bd"} Nov 03 21:25:49.482696 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:49.482595 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8" event={"ID":"e8b35668-9b5e-4b65-ac26-66f7311ffb01","Type":"ContainerStarted","Data":"53aea6bd30c4c9d7d4cd6fa636431ffaa084e523a0cc20b14f2e2e4ff87d152d"} Nov 03 21:25:49.482696 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:49.482677 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8" Nov 03 21:25:49.598969 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:49.598914 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t" podStartSLOduration=2.5989009 podStartE2EDuration="2.5989009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:49.59695271 +0000 UTC m=+3421.154943227" watchObservedRunningTime="2026-11-03 21:25:49.5989009 +0000 UTC m=+3421.156891399" Nov 03 21:25:50.102170 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:50.102111 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8" podStartSLOduration=2.102095189 podStartE2EDuration="2.102095189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:49.715504177 +0000 UTC m=+3421.273494689" watchObservedRunningTime="2026-11-03 21:25:50.102095189 +0000 UTC m=+3421.660085717" Nov 03 21:25:50.102541 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:50.102522 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q"] Nov 03 21:25:50.106067 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:50.106048 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q" Nov 03 21:25:50.160221 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:50.160196 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q"] Nov 03 21:25:50.171446 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:50.171423 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91210686-fd4d-44b3-a448-abf49f3961f7-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q\" (UID: \"91210686-fd4d-44b3-a448-abf49f3961f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q" Nov 03 21:25:50.171552 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:50.171452 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91210686-fd4d-44b3-a448-abf49f3961f7-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q\" (UID: \"91210686-fd4d-44b3-a448-abf49f3961f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q" Nov 03 21:25:50.171552 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:50.171497 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91210686-fd4d-44b3-a448-abf49f3961f7-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q\" (UID: \"91210686-fd4d-44b3-a448-abf49f3961f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q" Nov 03 21:25:50.171690 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:50.171663 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxz4l\" (UniqueName: \"kubernetes.io/projected/91210686-fd4d-44b3-a448-abf49f3961f7-kube-api-access-pxz4l\") pod \"maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q\" (UID: \"91210686-fd4d-44b3-a448-abf49f3961f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q" Nov 03 21:25:50.209224 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:50.209200 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-d9f39a-serving-cert\"" Nov 03 21:25:50.272812 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:50.272782 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91210686-fd4d-44b3-a448-abf49f3961f7-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q\" (UID: \"91210686-fd4d-44b3-a448-abf49f3961f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q" Nov 03 21:25:50.272988 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:50.272847 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxz4l\" (UniqueName: \"kubernetes.io/projected/91210686-fd4d-44b3-a448-abf49f3961f7-kube-api-access-pxz4l\") pod \"maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q\" (UID: \"91210686-fd4d-44b3-a448-abf49f3961f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q" Nov 03 21:25:50.272988 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:50.272916 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91210686-fd4d-44b3-a448-abf49f3961f7-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q\" (UID: \"91210686-fd4d-44b3-a448-abf49f3961f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q" Nov 03 21:25:50.272988 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:50.272933 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91210686-fd4d-44b3-a448-abf49f3961f7-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q\" (UID: \"91210686-fd4d-44b3-a448-abf49f3961f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q" Nov 03 21:25:50.273168 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:25:50.273047 4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-serving-cert: secret "maas-api-e2e-worker-single-main-d9f39a-serving-cert" not found Nov 03 21:25:50.273168 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:25:50.273107 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91210686-fd4d-44b3-a448-abf49f3961f7-maas-api-tls podName:91210686-fd4d-44b3-a448-abf49f3961f7 nodeName:}" failed. No retries permitted until 2026-11-03 21:25:50.773091435 +0000 UTC m=+3422.331081917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/91210686-fd4d-44b3-a448-abf49f3961f7-maas-api-tls") pod "maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q" (UID: "91210686-fd4d-44b3-a448-abf49f3961f7") : secret "maas-api-e2e-worker-single-main-d9f39a-serving-cert" not found Nov 03 21:25:50.273588 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:50.273571 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91210686-fd4d-44b3-a448-abf49f3961f7-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q\" (UID: \"91210686-fd4d-44b3-a448-abf49f3961f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q" Nov 03 21:25:50.275445 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:50.275422 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91210686-fd4d-44b3-a448-abf49f3961f7-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q\" (UID: \"91210686-fd4d-44b3-a448-abf49f3961f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q" Nov 03 21:25:50.332190 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:50.332160 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxz4l\" (UniqueName: \"kubernetes.io/projected/91210686-fd4d-44b3-a448-abf49f3961f7-kube-api-access-pxz4l\") pod \"maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q\" (UID: \"91210686-fd4d-44b3-a448-abf49f3961f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q" Nov 03 21:25:50.777702 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:50.777662 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91210686-fd4d-44b3-a448-abf49f3961f7-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q\" (UID: \"91210686-fd4d-44b3-a448-abf49f3961f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q" Nov 03 21:25:50.780287 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:50.780266 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91210686-fd4d-44b3-a448-abf49f3961f7-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q\" (UID: \"91210686-fd4d-44b3-a448-abf49f3961f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q" Nov 03 21:25:51.019671 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:51.019637 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q" Nov 03 21:25:51.391961 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:51.391934 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q"] Nov 03 21:25:51.394156 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:25:51.394124 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91210686_fd4d_44b3_a448_abf49f3961f7.slice/crio-d690837ef1fc399d9427d3333b1fc969e057c0d22f1b40e75feab80aa706bf0b WatchSource:0}: Error finding container d690837ef1fc399d9427d3333b1fc969e057c0d22f1b40e75feab80aa706bf0b: Status 404 returned error can't find the container with id d690837ef1fc399d9427d3333b1fc969e057c0d22f1b40e75feab80aa706bf0b Nov 03 21:25:51.395580 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:51.395560 4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:25:51.493865 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:51.493832 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q" event={"ID":"91210686-fd4d-44b3-a448-abf49f3961f7","Type":"ContainerStarted","Data":"d690837ef1fc399d9427d3333b1fc969e057c0d22f1b40e75feab80aa706bf0b"} Nov 03 21:25:51.743837 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:25:51.743753 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:25:52.500468 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:52.500430 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q" event={"ID":"91210686-fd4d-44b3-a448-abf49f3961f7","Type":"ContainerStarted","Data":"d8b4e1840b4cece3fff347d7582650308b531d67bbe3c573d642f7f618c83862"} Nov 03 21:25:52.500877 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:52.500528 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q" Nov 03 21:25:52.621707 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:52.621602 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q" podStartSLOduration=1.7099178080000002 podStartE2EDuration="2.621588643s" totalImagesPullingTime="911.670835ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:52.620454735 +0000 UTC m=+3424.178445233" watchObservedRunningTime="2026-11-03 21:25:52.621588643 +0000 UTC m=+3424.179579141" Nov 03 21:25:58.510688 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:25:58.510660 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q" Nov 03 21:26:00.510685 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:26:00.510648 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t" Nov 03 21:26:00.511119 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:26:00.510824 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8" Nov 03 21:26:08.576568 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:26:08.576537 4493 generic.go:356] "Generic (PLEG): container finished" podID="73ca984a-209a-431a-81bd-b04175901685" containerID="ad7914966cd99549772a54a490e79104cb02166093f47618c4a4982639a167d0" exitCode=0 Nov 03 21:26:08.576976 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:26:08.576613 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" event={"ID":"73ca984a-209a-431a-81bd-b04175901685","Type":"ContainerDied","Data":"ad7914966cd99549772a54a490e79104cb02166093f47618c4a4982639a167d0"} Nov 03 21:26:08.577024 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:26:08.576992 4493 scope.go:122] "RemoveContainer" containerID="ad7914966cd99549772a54a490e79104cb02166093f47618c4a4982639a167d0" Nov 03 21:26:08.578143 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:26:08.578117 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:26:08.578201 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:26:08.578185 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:26:08.578237 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:26:08.578219 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:26:09.590697 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:26:09.590644 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" event={"ID":"73ca984a-209a-431a-81bd-b04175901685","Type":"ContainerStarted","Data":"2acaa771eb26bc79205b510136ed820abf420774bfd5b189efc365c625abc04b"} Nov 03 21:26:10.473484 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:26:10.473450 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:26:10.474676 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:26:10.474652 4493 patch_prober.go:29] interesting pod/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 03 21:26:10.474800 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:26:10.474698 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" podUID="73ca984a-209a-431a-81bd-b04175901685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 03 21:26:11.473764 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:26:11.473730 4493 patch_prober.go:29] interesting pod/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 03 21:26:11.474186 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:26:11.473786 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" podUID="73ca984a-209a-431a-81bd-b04175901685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 03 21:26:12.444724 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:26:12.444691 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:26:12.474555 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:26:12.474516 4493 patch_prober.go:29] interesting pod/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 03 21:26:12.474957 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:26:12.474576 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" podUID="73ca984a-209a-431a-81bd-b04175901685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 03 21:26:13.473652 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:26:13.473618 4493 patch_prober.go:29] interesting pod/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 03 21:26:13.473835 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:26:13.473685 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" podUID="73ca984a-209a-431a-81bd-b04175901685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 03 21:26:14.474254 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:26:14.474220 4493 patch_prober.go:29] interesting pod/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 03 21:26:14.474677 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:26:14.474273 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" podUID="73ca984a-209a-431a-81bd-b04175901685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 03 21:26:15.477832 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:26:15.477798 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:26:15.478765 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:26:15.478743 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:26:21.743200 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:26:21.743159 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:26:51.743008 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:26:51.742975 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:27:20.466729 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:20.466690 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8"] Nov 03 21:27:20.575464 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:20.575425 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q"] Nov 03 21:27:20.575673 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:20.575611 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q" podUID="91210686-fd4d-44b3-a448-abf49f3961f7" containerName="maas-api" containerID="cri-o://d8b4e1840b4cece3fff347d7582650308b531d67bbe3c573d642f7f618c83862" gracePeriod=30 Nov 03 21:27:20.614988 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:20.614959 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t"] Nov 03 21:27:21.750018 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:27:21.749947 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:27:21.835732 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:21.835705 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q" Nov 03 21:27:21.926620 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:21.926552 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91210686-fd4d-44b3-a448-abf49f3961f7-kube-api-access-pxz4l\" (UniqueName: \"kubernetes.io/projected/91210686-fd4d-44b3-a448-abf49f3961f7-kube-api-access-pxz4l\") pod \"91210686-fd4d-44b3-a448-abf49f3961f7\" (UID: \"91210686-fd4d-44b3-a448-abf49f3961f7\") " Nov 03 21:27:21.926763 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:21.926658 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91210686-fd4d-44b3-a448-abf49f3961f7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91210686-fd4d-44b3-a448-abf49f3961f7-openshift-service-ca\") pod \"91210686-fd4d-44b3-a448-abf49f3961f7\" (UID: \"91210686-fd4d-44b3-a448-abf49f3961f7\") " Nov 03 21:27:21.926816 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:21.926800 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91210686-fd4d-44b3-a448-abf49f3961f7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91210686-fd4d-44b3-a448-abf49f3961f7-maas-api-tls\") pod \"91210686-fd4d-44b3-a448-abf49f3961f7\" (UID: \"91210686-fd4d-44b3-a448-abf49f3961f7\") " Nov 03 21:27:21.926855 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:21.926839 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91210686-fd4d-44b3-a448-abf49f3961f7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/91210686-fd4d-44b3-a448-abf49f3961f7-metrics-certs\") pod \"91210686-fd4d-44b3-a448-abf49f3961f7\" (UID: \"91210686-fd4d-44b3-a448-abf49f3961f7\") " Nov 03 21:27:21.927023 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:21.927001 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91210686-fd4d-44b3-a448-abf49f3961f7-openshift-service-ca" pod "91210686-fd4d-44b3-a448-abf49f3961f7" (UID: "91210686-fd4d-44b3-a448-abf49f3961f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:27:21.927165 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:21.927146 4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91210686-fd4d-44b3-a448-abf49f3961f7-openshift-service-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:21.929019 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:21.928993 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91210686-fd4d-44b3-a448-abf49f3961f7-kube-api-access-pxz4l" pod "91210686-fd4d-44b3-a448-abf49f3961f7" (UID: "91210686-fd4d-44b3-a448-abf49f3961f7"). InnerVolumeSpecName "kube-api-access-pxz4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:27:21.929019 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:21.928997 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91210686-fd4d-44b3-a448-abf49f3961f7-metrics-certs" pod "91210686-fd4d-44b3-a448-abf49f3961f7" (UID: "91210686-fd4d-44b3-a448-abf49f3961f7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:27:21.929150 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:21.929011 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91210686-fd4d-44b3-a448-abf49f3961f7-maas-api-tls" pod "91210686-fd4d-44b3-a448-abf49f3961f7" (UID: "91210686-fd4d-44b3-a448-abf49f3961f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:27:21.968321 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:21.968288 4493 generic.go:356] "Generic (PLEG): container finished" podID="91210686-fd4d-44b3-a448-abf49f3961f7" containerID="d8b4e1840b4cece3fff347d7582650308b531d67bbe3c573d642f7f618c83862" exitCode=0 Nov 03 21:27:21.968454 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:21.968392 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q" Nov 03 21:27:21.968454 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:21.968425 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q" event={"ID":"91210686-fd4d-44b3-a448-abf49f3961f7","Type":"ContainerDied","Data":"d8b4e1840b4cece3fff347d7582650308b531d67bbe3c573d642f7f618c83862"} Nov 03 21:27:21.968541 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:21.968476 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q" event={"ID":"91210686-fd4d-44b3-a448-abf49f3961f7","Type":"ContainerDied","Data":"d690837ef1fc399d9427d3333b1fc969e057c0d22f1b40e75feab80aa706bf0b"} Nov 03 21:27:21.968541 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:21.968497 4493 scope.go:122] "RemoveContainer" containerID="d8b4e1840b4cece3fff347d7582650308b531d67bbe3c573d642f7f618c83862" Nov 03 21:27:21.987719 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:21.987699 4493 scope.go:122] "RemoveContainer" containerID="d8b4e1840b4cece3fff347d7582650308b531d67bbe3c573d642f7f618c83862" Nov 03 21:27:21.988017 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:27:21.987996 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8b4e1840b4cece3fff347d7582650308b531d67bbe3c573d642f7f618c83862\": container with ID starting with d8b4e1840b4cece3fff347d7582650308b531d67bbe3c573d642f7f618c83862 not found: ID does not exist" containerID="d8b4e1840b4cece3fff347d7582650308b531d67bbe3c573d642f7f618c83862" Nov 03 21:27:21.988080 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:21.988024 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8b4e1840b4cece3fff347d7582650308b531d67bbe3c573d642f7f618c83862"} err="failed to get container status \"d8b4e1840b4cece3fff347d7582650308b531d67bbe3c573d642f7f618c83862\": rpc error: code = NotFound desc = could not find container \"d8b4e1840b4cece3fff347d7582650308b531d67bbe3c573d642f7f618c83862\": container with ID starting with d8b4e1840b4cece3fff347d7582650308b531d67bbe3c573d642f7f618c83862 not found: ID does not exist" Nov 03 21:27:22.027807 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:22.027781 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91210686-fd4d-44b3-a448-abf49f3961f7-maas-api-tls\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:22.027807 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:22.027805 4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91210686-fd4d-44b3-a448-abf49f3961f7-metrics-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:22.027983 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:22.027816 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxz4l\" (UniqueName: \"kubernetes.io/projected/91210686-fd4d-44b3-a448-abf49f3961f7-kube-api-access-pxz4l\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:22.136452 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:22.136418 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q"] Nov 03 21:27:22.140058 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:22.140032 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9f39a-68767f6944-vp88q"] Nov 03 21:27:22.561932 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:22.561899 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91210686-fd4d-44b3-a448-abf49f3961f7" path="/var/lib/kubelet/pods/91210686-fd4d-44b3-a448-abf49f3961f7/volumes" Nov 03 21:27:25.467151 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:25.467108 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8" podUID="e8b35668-9b5e-4b65-ac26-66f7311ffb01" containerName="payload-processing" containerID="cri-o://e9ff42ff5f4e8c100d72e89092473bd5c7c5a533c5715cf6f4b3a41cd26dc1bd" gracePeriod=25 Nov 03 21:27:25.615899 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:25.615859 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t" podUID="b38bde3b-9a8d-4907-aa35-29287d53353f" containerName="payload-pre-processing" containerID="cri-o://726cdeed5e0ea427a93305bfbded79a8bb7575e5d96bf0ce6f3f55399f8331d8" gracePeriod=25 Nov 03 21:27:30.489660 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:30.489613 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t" podUID="b38bde3b-9a8d-4907-aa35-29287d53353f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:27:30.490058 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:30.489635 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8" podUID="e8b35668-9b5e-4b65-ac26-66f7311ffb01" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:27:31.734194 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:31.734152 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf"] Nov 03 21:27:31.734694 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:31.734422 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" podUID="73ca984a-209a-431a-81bd-b04175901685" containerName="istio-proxy" containerID="cri-o://2acaa771eb26bc79205b510136ed820abf420774bfd5b189efc365c625abc04b" gracePeriod=30 Nov 03 21:27:36.994054 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:36.994024 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:27:37.045798 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.045770 4493 generic.go:356] "Generic (PLEG): container finished" podID="73ca984a-209a-431a-81bd-b04175901685" containerID="2acaa771eb26bc79205b510136ed820abf420774bfd5b189efc365c625abc04b" exitCode=0 Nov 03 21:27:37.045938 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.045828 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" Nov 03 21:27:37.045938 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.045859 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" event={"ID":"73ca984a-209a-431a-81bd-b04175901685","Type":"ContainerDied","Data":"2acaa771eb26bc79205b510136ed820abf420774bfd5b189efc365c625abc04b"} Nov 03 21:27:37.045938 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.045918 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf" event={"ID":"73ca984a-209a-431a-81bd-b04175901685","Type":"ContainerDied","Data":"70ca31905a8a04c792e221ff07e3b306ac9b72831d89b7e21b5ed73775db5cc5"} Nov 03 21:27:37.046043 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.045939 4493 scope.go:122] "RemoveContainer" containerID="2acaa771eb26bc79205b510136ed820abf420774bfd5b189efc365c625abc04b" Nov 03 21:27:37.054916 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.054897 4493 scope.go:122] "RemoveContainer" containerID="ad7914966cd99549772a54a490e79104cb02166093f47618c4a4982639a167d0" Nov 03 21:27:37.063248 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.063230 4493 scope.go:122] "RemoveContainer" containerID="2acaa771eb26bc79205b510136ed820abf420774bfd5b189efc365c625abc04b" Nov 03 21:27:37.063568 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:27:37.063548 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2acaa771eb26bc79205b510136ed820abf420774bfd5b189efc365c625abc04b\": container with ID starting with 2acaa771eb26bc79205b510136ed820abf420774bfd5b189efc365c625abc04b not found: ID does not exist" containerID="2acaa771eb26bc79205b510136ed820abf420774bfd5b189efc365c625abc04b" Nov 03 21:27:37.063666 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.063582 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2acaa771eb26bc79205b510136ed820abf420774bfd5b189efc365c625abc04b"} err="failed to get container status \"2acaa771eb26bc79205b510136ed820abf420774bfd5b189efc365c625abc04b\": rpc error: code = NotFound desc = could not find container \"2acaa771eb26bc79205b510136ed820abf420774bfd5b189efc365c625abc04b\": container with ID starting with 2acaa771eb26bc79205b510136ed820abf420774bfd5b189efc365c625abc04b not found: ID does not exist" Nov 03 21:27:37.063666 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.063614 4493 scope.go:122] "RemoveContainer" containerID="ad7914966cd99549772a54a490e79104cb02166093f47618c4a4982639a167d0" Nov 03 21:27:37.063895 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:27:37.063877 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad7914966cd99549772a54a490e79104cb02166093f47618c4a4982639a167d0\": container with ID starting with ad7914966cd99549772a54a490e79104cb02166093f47618c4a4982639a167d0 not found: ID does not exist" containerID="ad7914966cd99549772a54a490e79104cb02166093f47618c4a4982639a167d0" Nov 03 21:27:37.063934 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.063902 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad7914966cd99549772a54a490e79104cb02166093f47618c4a4982639a167d0"} err="failed to get container status \"ad7914966cd99549772a54a490e79104cb02166093f47618c4a4982639a167d0\": rpc error: code = NotFound desc = could not find container \"ad7914966cd99549772a54a490e79104cb02166093f47618c4a4982639a167d0\": container with ID starting with ad7914966cd99549772a54a490e79104cb02166093f47618c4a4982639a167d0 not found: ID does not exist" Nov 03 21:27:37.071264 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.071245 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73ca984a-209a-431a-81bd-b04175901685-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73ca984a-209a-431a-81bd-b04175901685-istiod-ca-cert\") pod \"73ca984a-209a-431a-81bd-b04175901685\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " Nov 03 21:27:37.071366 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.071278 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-workload-certs\") pod \"73ca984a-209a-431a-81bd-b04175901685\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " Nov 03 21:27:37.071366 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.071308 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-istio-data\") pod \"73ca984a-209a-431a-81bd-b04175901685\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " Nov 03 21:27:37.071489 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.071379 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-credential-socket\") pod \"73ca984a-209a-431a-81bd-b04175901685\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " Nov 03 21:27:37.071489 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.071402 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-istio-envoy\") pod \"73ca984a-209a-431a-81bd-b04175901685\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " Nov 03 21:27:37.071489 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.071428 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-workload-socket\") pod \"73ca984a-209a-431a-81bd-b04175901685\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " Nov 03 21:27:37.071489 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.071459 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ca984a-209a-431a-81bd-b04175901685-kube-api-access-z9ctf\" (UniqueName: \"kubernetes.io/projected/73ca984a-209a-431a-81bd-b04175901685-kube-api-access-z9ctf\") pod \"73ca984a-209a-431a-81bd-b04175901685\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " Nov 03 21:27:37.071489 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.071485 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ca984a-209a-431a-81bd-b04175901685-istio-token\" (UniqueName: \"kubernetes.io/projected/73ca984a-209a-431a-81bd-b04175901685-istio-token\") pod \"73ca984a-209a-431a-81bd-b04175901685\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " Nov 03 21:27:37.071759 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.071524 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73ca984a-209a-431a-81bd-b04175901685-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73ca984a-209a-431a-81bd-b04175901685-istio-podinfo\") pod \"73ca984a-209a-431a-81bd-b04175901685\" (UID: \"73ca984a-209a-431a-81bd-b04175901685\") " Nov 03 21:27:37.071856 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.071522 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-workload-certs" pod "73ca984a-209a-431a-81bd-b04175901685" (UID: "73ca984a-209a-431a-81bd-b04175901685"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:27:37.071856 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.071624 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-istio-data" pod "73ca984a-209a-431a-81bd-b04175901685" (UID: "73ca984a-209a-431a-81bd-b04175901685"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:27:37.071856 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.071637 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-credential-socket" pod "73ca984a-209a-431a-81bd-b04175901685" (UID: "73ca984a-209a-431a-81bd-b04175901685"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:27:37.072054 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.071747 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-workload-socket" pod "73ca984a-209a-431a-81bd-b04175901685" (UID: "73ca984a-209a-431a-81bd-b04175901685"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:27:37.072054 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.071754 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ca984a-209a-431a-81bd-b04175901685-istiod-ca-cert" pod "73ca984a-209a-431a-81bd-b04175901685" (UID: "73ca984a-209a-431a-81bd-b04175901685"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:27:37.072054 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.071960 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73ca984a-209a-431a-81bd-b04175901685-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:37.072054 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.071975 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:37.072054 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.071983 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:37.072054 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.071992 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:37.072054 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.072000 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:37.073790 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.073765 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73ca984a-209a-431a-81bd-b04175901685-istio-podinfo" pod "73ca984a-209a-431a-81bd-b04175901685" (UID: "73ca984a-209a-431a-81bd-b04175901685"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:27:37.073869 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.073816 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-istio-envoy" pod "73ca984a-209a-431a-81bd-b04175901685" (UID: "73ca984a-209a-431a-81bd-b04175901685"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:27:37.073869 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.073857 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ca984a-209a-431a-81bd-b04175901685-istio-token" pod "73ca984a-209a-431a-81bd-b04175901685" (UID: "73ca984a-209a-431a-81bd-b04175901685"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:27:37.073869 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.073852 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ca984a-209a-431a-81bd-b04175901685-kube-api-access-z9ctf" pod "73ca984a-209a-431a-81bd-b04175901685" (UID: "73ca984a-209a-431a-81bd-b04175901685"). InnerVolumeSpecName "kube-api-access-z9ctf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:27:37.172940 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.172911 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73ca984a-209a-431a-81bd-b04175901685-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:37.172940 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.172940 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9ctf\" (UniqueName: \"kubernetes.io/projected/73ca984a-209a-431a-81bd-b04175901685-kube-api-access-z9ctf\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:37.173120 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.172951 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73ca984a-209a-431a-81bd-b04175901685-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:37.173120 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.172959 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73ca984a-209a-431a-81bd-b04175901685-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:37.468853 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.468822 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf"] Nov 03 21:27:37.474669 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:37.474646 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d9f39a-openshift-default-587645d889t4xrf"] Nov 03 21:27:38.560411 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:38.560379 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73ca984a-209a-431a-81bd-b04175901685" path="/var/lib/kubelet/pods/73ca984a-209a-431a-81bd-b04175901685/volumes" Nov 03 21:27:40.489296 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:40.489257 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t" podUID="b38bde3b-9a8d-4907-aa35-29287d53353f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:27:40.489805 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:40.489257 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8" podUID="e8b35668-9b5e-4b65-ac26-66f7311ffb01" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:27:45.495803 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:27:45.495773 4493 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b35668_9b5e_4b65_ac26_66f7311ffb01.slice/crio-e9ff42ff5f4e8c100d72e89092473bd5c7c5a533c5715cf6f4b3a41cd26dc1bd.scope\": RecentStats: unable to find data in memory cache]" Nov 03 21:27:45.496186 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:27:45.495791 4493 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b35668_9b5e_4b65_ac26_66f7311ffb01.slice/crio-conmon-e9ff42ff5f4e8c100d72e89092473bd5c7c5a533c5715cf6f4b3a41cd26dc1bd.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b35668_9b5e_4b65_ac26_66f7311ffb01.slice/crio-e9ff42ff5f4e8c100d72e89092473bd5c7c5a533c5715cf6f4b3a41cd26dc1bd.scope\": RecentStats: unable to find data in memory cache]" Nov 03 21:27:45.741808 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:45.741783 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8" Nov 03 21:27:45.854527 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:45.854476 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8b35668-9b5e-4b65-ac26-66f7311ffb01-kube-api-access-jqttb\" (UniqueName: \"kubernetes.io/projected/e8b35668-9b5e-4b65-ac26-66f7311ffb01-kube-api-access-jqttb\") pod \"e8b35668-9b5e-4b65-ac26-66f7311ffb01\" (UID: \"e8b35668-9b5e-4b65-ac26-66f7311ffb01\") " Nov 03 21:27:45.854710 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:45.854633 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8b35668-9b5e-4b65-ac26-66f7311ffb01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8b35668-9b5e-4b65-ac26-66f7311ffb01-plugins-config-volume\") pod \"e8b35668-9b5e-4b65-ac26-66f7311ffb01\" (UID: \"e8b35668-9b5e-4b65-ac26-66f7311ffb01\") " Nov 03 21:27:45.854995 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:45.854952 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8b35668-9b5e-4b65-ac26-66f7311ffb01-plugins-config-volume" pod "e8b35668-9b5e-4b65-ac26-66f7311ffb01" (UID: "e8b35668-9b5e-4b65-ac26-66f7311ffb01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:27:45.858580 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:45.858518 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8b35668-9b5e-4b65-ac26-66f7311ffb01-kube-api-access-jqttb" pod "e8b35668-9b5e-4b65-ac26-66f7311ffb01" (UID: "e8b35668-9b5e-4b65-ac26-66f7311ffb01"). InnerVolumeSpecName "kube-api-access-jqttb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:27:45.883566 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:45.883544 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t" Nov 03 21:27:45.955807 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:45.955783 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b38bde3b-9a8d-4907-aa35-29287d53353f-kube-api-access-tg6zk\" (UniqueName: \"kubernetes.io/projected/b38bde3b-9a8d-4907-aa35-29287d53353f-kube-api-access-tg6zk\") pod \"b38bde3b-9a8d-4907-aa35-29287d53353f\" (UID: \"b38bde3b-9a8d-4907-aa35-29287d53353f\") " Nov 03 21:27:45.955952 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:45.955838 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b38bde3b-9a8d-4907-aa35-29287d53353f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38bde3b-9a8d-4907-aa35-29287d53353f-plugins-config-volume\") pod \"b38bde3b-9a8d-4907-aa35-29287d53353f\" (UID: \"b38bde3b-9a8d-4907-aa35-29287d53353f\") " Nov 03 21:27:45.956166 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:45.956139 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqttb\" (UniqueName: \"kubernetes.io/projected/e8b35668-9b5e-4b65-ac26-66f7311ffb01-kube-api-access-jqttb\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:45.956270 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:45.956169 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8b35668-9b5e-4b65-ac26-66f7311ffb01-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:45.956270 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:45.956236 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b38bde3b-9a8d-4907-aa35-29287d53353f-plugins-config-volume" pod "b38bde3b-9a8d-4907-aa35-29287d53353f" (UID: "b38bde3b-9a8d-4907-aa35-29287d53353f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:27:45.957966 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:45.957944 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b38bde3b-9a8d-4907-aa35-29287d53353f-kube-api-access-tg6zk" pod "b38bde3b-9a8d-4907-aa35-29287d53353f" (UID: "b38bde3b-9a8d-4907-aa35-29287d53353f"). InnerVolumeSpecName "kube-api-access-tg6zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:27:46.057564 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:46.057537 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38bde3b-9a8d-4907-aa35-29287d53353f-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:46.057564 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:46.057564 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg6zk\" (UniqueName: \"kubernetes.io/projected/b38bde3b-9a8d-4907-aa35-29287d53353f-kube-api-access-tg6zk\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:46.090503 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:46.090473 4493 generic.go:356] "Generic (PLEG): container finished" podID="b38bde3b-9a8d-4907-aa35-29287d53353f" containerID="726cdeed5e0ea427a93305bfbded79a8bb7575e5d96bf0ce6f3f55399f8331d8" exitCode=0 Nov 03 21:27:46.090641 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:46.090521 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t" Nov 03 21:27:46.090641 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:46.090560 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t" event={"ID":"b38bde3b-9a8d-4907-aa35-29287d53353f","Type":"ContainerDied","Data":"726cdeed5e0ea427a93305bfbded79a8bb7575e5d96bf0ce6f3f55399f8331d8"} Nov 03 21:27:46.090745 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:46.090647 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t" event={"ID":"b38bde3b-9a8d-4907-aa35-29287d53353f","Type":"ContainerDied","Data":"9929b9cc016fd79e39a6defc0c5d0e90a770c4f1bad00af86dda187508dfc79c"} Nov 03 21:27:46.090745 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:46.090675 4493 scope.go:122] "RemoveContainer" containerID="726cdeed5e0ea427a93305bfbded79a8bb7575e5d96bf0ce6f3f55399f8331d8" Nov 03 21:27:46.091956 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:46.091935 4493 generic.go:356] "Generic (PLEG): container finished" podID="e8b35668-9b5e-4b65-ac26-66f7311ffb01" containerID="e9ff42ff5f4e8c100d72e89092473bd5c7c5a533c5715cf6f4b3a41cd26dc1bd" exitCode=0 Nov 03 21:27:46.092032 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:46.091964 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8" event={"ID":"e8b35668-9b5e-4b65-ac26-66f7311ffb01","Type":"ContainerDied","Data":"e9ff42ff5f4e8c100d72e89092473bd5c7c5a533c5715cf6f4b3a41cd26dc1bd"} Nov 03 21:27:46.092032 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:46.091996 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8" Nov 03 21:27:46.092032 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:46.092005 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8" event={"ID":"e8b35668-9b5e-4b65-ac26-66f7311ffb01","Type":"ContainerDied","Data":"53aea6bd30c4c9d7d4cd6fa636431ffaa084e523a0cc20b14f2e2e4ff87d152d"} Nov 03 21:27:46.101999 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:46.101937 4493 scope.go:122] "RemoveContainer" containerID="726cdeed5e0ea427a93305bfbded79a8bb7575e5d96bf0ce6f3f55399f8331d8" Nov 03 21:27:46.102413 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:27:46.102390 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"726cdeed5e0ea427a93305bfbded79a8bb7575e5d96bf0ce6f3f55399f8331d8\": container with ID starting with 726cdeed5e0ea427a93305bfbded79a8bb7575e5d96bf0ce6f3f55399f8331d8 not found: ID does not exist" containerID="726cdeed5e0ea427a93305bfbded79a8bb7575e5d96bf0ce6f3f55399f8331d8" Nov 03 21:27:46.102530 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:46.102425 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"726cdeed5e0ea427a93305bfbded79a8bb7575e5d96bf0ce6f3f55399f8331d8"} err="failed to get container status \"726cdeed5e0ea427a93305bfbded79a8bb7575e5d96bf0ce6f3f55399f8331d8\": rpc error: code = NotFound desc = could not find container \"726cdeed5e0ea427a93305bfbded79a8bb7575e5d96bf0ce6f3f55399f8331d8\": container with ID starting with 726cdeed5e0ea427a93305bfbded79a8bb7575e5d96bf0ce6f3f55399f8331d8 not found: ID does not exist" Nov 03 21:27:46.102530 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:46.102449 4493 scope.go:122] "RemoveContainer" containerID="e9ff42ff5f4e8c100d72e89092473bd5c7c5a533c5715cf6f4b3a41cd26dc1bd" Nov 03 21:27:46.112182 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:46.112162 4493 scope.go:122] "RemoveContainer" containerID="e9ff42ff5f4e8c100d72e89092473bd5c7c5a533c5715cf6f4b3a41cd26dc1bd" Nov 03 21:27:46.112492 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:27:46.112471 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9ff42ff5f4e8c100d72e89092473bd5c7c5a533c5715cf6f4b3a41cd26dc1bd\": container with ID starting with e9ff42ff5f4e8c100d72e89092473bd5c7c5a533c5715cf6f4b3a41cd26dc1bd not found: ID does not exist" containerID="e9ff42ff5f4e8c100d72e89092473bd5c7c5a533c5715cf6f4b3a41cd26dc1bd" Nov 03 21:27:46.112555 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:46.112500 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ff42ff5f4e8c100d72e89092473bd5c7c5a533c5715cf6f4b3a41cd26dc1bd"} err="failed to get container status \"e9ff42ff5f4e8c100d72e89092473bd5c7c5a533c5715cf6f4b3a41cd26dc1bd\": rpc error: code = NotFound desc = could not find container \"e9ff42ff5f4e8c100d72e89092473bd5c7c5a533c5715cf6f4b3a41cd26dc1bd\": container with ID starting with e9ff42ff5f4e8c100d72e89092473bd5c7c5a533c5715cf6f4b3a41cd26dc1bd not found: ID does not exist" Nov 03 21:27:46.258905 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:46.258877 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t"] Nov 03 21:27:46.263237 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:46.263211 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9f39a-78786h2c6t"] Nov 03 21:27:46.376138 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:46.376050 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8"] Nov 03 21:27:46.380082 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:46.380060 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d9f39a-8d7579659rbld8"] Nov 03 21:27:46.561181 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:46.561143 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b38bde3b-9a8d-4907-aa35-29287d53353f" path="/var/lib/kubelet/pods/b38bde3b-9a8d-4907-aa35-29287d53353f/volumes" Nov 03 21:27:46.561562 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:27:46.561546 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8b35668-9b5e-4b65-ac26-66f7311ffb01" path="/var/lib/kubelet/pods/e8b35668-9b5e-4b65-ac26-66f7311ffb01/volumes" Nov 03 21:27:51.748770 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:27:51.748736 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:28:21.744558 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:28:21.744519 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:28:49.262736 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:28:49.262711 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 21:28:49.349544 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:28:49.349520 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 21:28:51.744059 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:28:51.744020 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:29:21.744772 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:29:21.744734 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:29:51.743746 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:29:51.743711 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:30:18.056312 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.056274 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8"] Nov 03 21:30:18.056827 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.056737 4493 cpu_manager.go:414] "containerMap: removing container" podUID="73ca984a-209a-431a-81bd-b04175901685" containerName="istio-proxy" Nov 03 21:30:18.056827 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.056750 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ca984a-209a-431a-81bd-b04175901685" containerName="istio-proxy" Nov 03 21:30:18.056827 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.056762 4493 cpu_manager.go:414] "containerMap: removing container" podUID="b38bde3b-9a8d-4907-aa35-29287d53353f" containerName="payload-pre-processing" Nov 03 21:30:18.056827 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.056767 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b38bde3b-9a8d-4907-aa35-29287d53353f" containerName="payload-pre-processing" Nov 03 21:30:18.056827 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.056786 4493 cpu_manager.go:414] "containerMap: removing container" podUID="73ca984a-209a-431a-81bd-b04175901685" containerName="istio-proxy" Nov 03 21:30:18.056827 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.056791 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ca984a-209a-431a-81bd-b04175901685" containerName="istio-proxy" Nov 03 21:30:18.056827 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.056799 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e8b35668-9b5e-4b65-ac26-66f7311ffb01" containerName="payload-processing" Nov 03 21:30:18.056827 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.056804 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8b35668-9b5e-4b65-ac26-66f7311ffb01" containerName="payload-processing" Nov 03 21:30:18.056827 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.056816 4493 cpu_manager.go:414] "containerMap: removing container" podUID="91210686-fd4d-44b3-a448-abf49f3961f7" containerName="maas-api" Nov 03 21:30:18.056827 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.056820 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91210686-fd4d-44b3-a448-abf49f3961f7" containerName="maas-api" Nov 03 21:30:18.057151 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.056887 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ca984a-209a-431a-81bd-b04175901685" containerName="istio-proxy" Nov 03 21:30:18.057151 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.056895 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b38bde3b-9a8d-4907-aa35-29287d53353f" containerName="payload-pre-processing" Nov 03 21:30:18.057151 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.056903 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8b35668-9b5e-4b65-ac26-66f7311ffb01" containerName="payload-processing" Nov 03 21:30:18.057151 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.056912 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="91210686-fd4d-44b3-a448-abf49f3961f7" containerName="maas-api" Nov 03 21:30:18.060110 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.060093 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.122080 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.122049 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8"] Nov 03 21:30:18.165300 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.165262 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4srkb\" (UniqueName: \"kubernetes.io/projected/01cb721e-c861-49ed-a186-db4cab738a85-kube-api-access-4srkb\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.165481 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.165303 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-istio-envoy\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.165481 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.165462 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01cb721e-c861-49ed-a186-db4cab738a85-istio-podinfo\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.165561 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.165502 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01cb721e-c861-49ed-a186-db4cab738a85-istio-token\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.165599 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.165570 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-workload-socket\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.165599 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.165590 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-istio-data\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.165671 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.165614 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01cb721e-c861-49ed-a186-db4cab738a85-istiod-ca-cert\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.165713 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.165699 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-credential-socket\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.165750 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.165720 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-workload-certs\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.189261 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.189235 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-6d8154-openshift-default-dockercfg-6tvjl\"" Nov 03 21:30:18.266721 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.266696 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01cb721e-c861-49ed-a186-db4cab738a85-istiod-ca-cert\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.266872 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.266748 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-credential-socket\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.266872 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.266828 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-workload-certs\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.266872 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.266861 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4srkb\" (UniqueName: \"kubernetes.io/projected/01cb721e-c861-49ed-a186-db4cab738a85-kube-api-access-4srkb\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.267041 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.266897 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-istio-envoy\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.267041 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.266946 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01cb721e-c861-49ed-a186-db4cab738a85-istio-podinfo\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.267041 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.266987 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01cb721e-c861-49ed-a186-db4cab738a85-istio-token\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.267244 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.267210 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-credential-socket\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.267373 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.267249 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-workload-certs\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.267444 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.267378 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-workload-socket\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.267444 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.267407 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-istio-data\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.267542 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.267478 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01cb721e-c861-49ed-a186-db4cab738a85-istiod-ca-cert\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.267748 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.267689 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-workload-socket\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.267748 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.267720 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-istio-data\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.269462 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.269440 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-istio-envoy\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.269563 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.269545 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01cb721e-c861-49ed-a186-db4cab738a85-istio-podinfo\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.326450 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.326372 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01cb721e-c861-49ed-a186-db4cab738a85-istio-token\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.326592 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.326542 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4srkb\" (UniqueName: \"kubernetes.io/projected/01cb721e-c861-49ed-a186-db4cab738a85-kube-api-access-4srkb\") pod \"e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.372183 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.372149 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:18.822866 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.822837 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8"] Nov 03 21:30:18.823914 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:30:18.823885 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01cb721e_c861_49ed_a186_db4cab738a85.slice/crio-c685f2f8b3ce0d19a359b2d727895dad4268372f3177b328a8d7dbfc29cf55ca WatchSource:0}: Error finding container c685f2f8b3ce0d19a359b2d727895dad4268372f3177b328a8d7dbfc29cf55ca: Status 404 returned error can't find the container with id c685f2f8b3ce0d19a359b2d727895dad4268372f3177b328a8d7dbfc29cf55ca Nov 03 21:30:18.826126 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.826097 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:30:18.826207 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.826157 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:30:18.826207 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.826185 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:30:18.859118 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:18.859092 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" event={"ID":"01cb721e-c861-49ed-a186-db4cab738a85","Type":"ContainerStarted","Data":"c685f2f8b3ce0d19a359b2d727895dad4268372f3177b328a8d7dbfc29cf55ca"} Nov 03 21:30:19.865998 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:19.865960 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" event={"ID":"01cb721e-c861-49ed-a186-db4cab738a85","Type":"ContainerStarted","Data":"2d95b24fc7bc20d6bc38b461f4f4a6cd7475ef6ca8e16d0cbad537f0a06c3fb8"} Nov 03 21:30:20.047455 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:20.047400 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" podStartSLOduration=2.047387658 podStartE2EDuration="2.047387658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:20.035943851 +0000 UTC m=+3691.593934347" watchObservedRunningTime="2026-11-03 21:30:20.047387658 +0000 UTC m=+3691.605378158" Nov 03 21:30:20.372492 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:20.372462 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:20.377555 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:20.377524 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:20.873642 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:20.873605 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:20.874495 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:20.874476 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:21.744072 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:30:21.744037 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:30:26.474193 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:26.474155 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9"] Nov 03 21:30:26.474834 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:26.474812 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ca984a-209a-431a-81bd-b04175901685" containerName="istio-proxy" Nov 03 21:30:26.478017 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:26.477997 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9" Nov 03 21:30:26.530832 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:26.530803 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9"] Nov 03 21:30:26.551048 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:26.551023 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/257fe914-4f27-46be-8f87-d4c0376833c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9\" (UID: \"257fe914-4f27-46be-8f87-d4c0376833c6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9" Nov 03 21:30:26.551185 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:26.551078 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7jpp\" (UniqueName: \"kubernetes.io/projected/257fe914-4f27-46be-8f87-d4c0376833c6-kube-api-access-s7jpp\") pod \"payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9\" (UID: \"257fe914-4f27-46be-8f87-d4c0376833c6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9" Nov 03 21:30:26.580627 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:26.580567 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-6d8154\"" Nov 03 21:30:26.622206 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:26.622174 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88"] Nov 03 21:30:26.625813 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:26.625788 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88" Nov 03 21:30:26.652013 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:26.651982 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/257fe914-4f27-46be-8f87-d4c0376833c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9\" (UID: \"257fe914-4f27-46be-8f87-d4c0376833c6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9" Nov 03 21:30:26.652138 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:26.652040 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7jpp\" (UniqueName: \"kubernetes.io/projected/257fe914-4f27-46be-8f87-d4c0376833c6-kube-api-access-s7jpp\") pod \"payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9\" (UID: \"257fe914-4f27-46be-8f87-d4c0376833c6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9" Nov 03 21:30:26.652757 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:26.652736 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/257fe914-4f27-46be-8f87-d4c0376833c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9\" (UID: \"257fe914-4f27-46be-8f87-d4c0376833c6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9" Nov 03 21:30:26.679660 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:26.679633 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-abee75d5-dockercfg-zdpj8\"" Nov 03 21:30:26.679888 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:26.679858 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88"] Nov 03 21:30:26.710393 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:26.710354 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7jpp\" (UniqueName: \"kubernetes.io/projected/257fe914-4f27-46be-8f87-d4c0376833c6-kube-api-access-s7jpp\") pod \"payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9\" (UID: \"257fe914-4f27-46be-8f87-d4c0376833c6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9" Nov 03 21:30:26.752592 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:26.752516 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxrgb\" (UniqueName: \"kubernetes.io/projected/ebd6b058-cf2b-444b-97c3-776ccb3d4776-kube-api-access-wxrgb\") pod \"payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88\" (UID: \"ebd6b058-cf2b-444b-97c3-776ccb3d4776\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88" Nov 03 21:30:26.752714 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:26.752639 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd6b058-cf2b-444b-97c3-776ccb3d4776-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88\" (UID: \"ebd6b058-cf2b-444b-97c3-776ccb3d4776\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88" Nov 03 21:30:26.791514 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:26.791488 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9" Nov 03 21:30:26.853936 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:26.853901 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd6b058-cf2b-444b-97c3-776ccb3d4776-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88\" (UID: \"ebd6b058-cf2b-444b-97c3-776ccb3d4776\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88" Nov 03 21:30:26.854070 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:26.854019 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxrgb\" (UniqueName: \"kubernetes.io/projected/ebd6b058-cf2b-444b-97c3-776ccb3d4776-kube-api-access-wxrgb\") pod \"payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88\" (UID: \"ebd6b058-cf2b-444b-97c3-776ccb3d4776\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88" Nov 03 21:30:26.855325 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:26.855292 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd6b058-cf2b-444b-97c3-776ccb3d4776-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88\" (UID: \"ebd6b058-cf2b-444b-97c3-776ccb3d4776\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88" Nov 03 21:30:26.912666 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:26.912636 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxrgb\" (UniqueName: \"kubernetes.io/projected/ebd6b058-cf2b-444b-97c3-776ccb3d4776-kube-api-access-wxrgb\") pod \"payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88\" (UID: \"ebd6b058-cf2b-444b-97c3-776ccb3d4776\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88" Nov 03 21:30:26.938638 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:26.938608 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88" Nov 03 21:30:27.165748 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:27.165720 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9"] Nov 03 21:30:27.166403 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:30:27.166382 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257fe914_4f27_46be_8f87_d4c0376833c6.slice/crio-dc079e94a1c15d2326ede398199e6465359ad81d340918de84a86bb33cefcc4f WatchSource:0}: Error finding container dc079e94a1c15d2326ede398199e6465359ad81d340918de84a86bb33cefcc4f: Status 404 returned error can't find the container with id dc079e94a1c15d2326ede398199e6465359ad81d340918de84a86bb33cefcc4f Nov 03 21:30:27.313472 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:27.313445 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88"] Nov 03 21:30:27.314356 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:30:27.314315 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd6b058_cf2b_444b_97c3_776ccb3d4776.slice/crio-d16ea5f5d1d2b896df9637916a16dbbf77e8036836a123ca944b876cf745e3c9 WatchSource:0}: Error finding container d16ea5f5d1d2b896df9637916a16dbbf77e8036836a123ca944b876cf745e3c9: Status 404 returned error can't find the container with id d16ea5f5d1d2b896df9637916a16dbbf77e8036836a123ca944b876cf745e3c9 Nov 03 21:30:27.914060 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:27.914024 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88" event={"ID":"ebd6b058-cf2b-444b-97c3-776ccb3d4776","Type":"ContainerStarted","Data":"a09ac51bec4576ee48d867948b99a340e4048bb428efb6604d77dc647b00970f"} Nov 03 21:30:27.914496 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:27.914077 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88" event={"ID":"ebd6b058-cf2b-444b-97c3-776ccb3d4776","Type":"ContainerStarted","Data":"d16ea5f5d1d2b896df9637916a16dbbf77e8036836a123ca944b876cf745e3c9"} Nov 03 21:30:27.914496 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:27.914129 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88" Nov 03 21:30:27.915806 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:27.915784 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9" event={"ID":"257fe914-4f27-46be-8f87-d4c0376833c6","Type":"ContainerStarted","Data":"fd87ab1be70fa78bcfd3c2110886e4db9645628bc9caaaa0540d8bda9065be37"} Nov 03 21:30:27.915909 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:27.915820 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9" event={"ID":"257fe914-4f27-46be-8f87-d4c0376833c6","Type":"ContainerStarted","Data":"dc079e94a1c15d2326ede398199e6465359ad81d340918de84a86bb33cefcc4f"} Nov 03 21:30:27.915909 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:27.915880 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9" Nov 03 21:30:28.033839 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:28.033787 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88" podStartSLOduration=2.033775191 podStartE2EDuration="2.033775191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:28.031944561 +0000 UTC m=+3699.589935059" watchObservedRunningTime="2026-11-03 21:30:28.033775191 +0000 UTC m=+3699.591765689" Nov 03 21:30:28.151166 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:28.151108 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9" podStartSLOduration=2.151088483 podStartE2EDuration="2.151088483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:28.147317816 +0000 UTC m=+3699.705308341" watchObservedRunningTime="2026-11-03 21:30:28.151088483 +0000 UTC m=+3699.709078982" Nov 03 21:30:38.932796 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:38.932766 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88" Nov 03 21:30:38.933214 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:38.932891 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9" Nov 03 21:30:46.024974 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:46.024939 4493 generic.go:356] "Generic (PLEG): container finished" podID="01cb721e-c861-49ed-a186-db4cab738a85" containerID="2d95b24fc7bc20d6bc38b461f4f4a6cd7475ef6ca8e16d0cbad537f0a06c3fb8" exitCode=0 Nov 03 21:30:46.025422 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:46.025007 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" event={"ID":"01cb721e-c861-49ed-a186-db4cab738a85","Type":"ContainerDied","Data":"2d95b24fc7bc20d6bc38b461f4f4a6cd7475ef6ca8e16d0cbad537f0a06c3fb8"} Nov 03 21:30:46.025422 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:46.025415 4493 scope.go:122] "RemoveContainer" containerID="2d95b24fc7bc20d6bc38b461f4f4a6cd7475ef6ca8e16d0cbad537f0a06c3fb8" Nov 03 21:30:46.026641 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:46.026612 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:30:46.026731 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:46.026686 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:30:46.026731 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:46.026717 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:30:47.033782 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:47.033747 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" event={"ID":"01cb721e-c861-49ed-a186-db4cab738a85","Type":"ContainerStarted","Data":"964683f45e79203be1fc8561ada9e887a7dc30ad4a94ab8db1a6206dd0b07784"} Nov 03 21:30:47.372435 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:47.372390 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:47.373938 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:47.373904 4493 patch_prober.go:29] interesting pod/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 03 21:30:47.374124 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:47.373968 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" podUID="01cb721e-c861-49ed-a186-db4cab738a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 03 21:30:48.372537 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:48.372503 4493 patch_prober.go:29] interesting pod/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 03 21:30:48.372948 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:48.372558 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" podUID="01cb721e-c861-49ed-a186-db4cab738a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 03 21:30:49.373443 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:49.373409 4493 patch_prober.go:29] interesting pod/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 03 21:30:49.373991 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:49.373471 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" podUID="01cb721e-c861-49ed-a186-db4cab738a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 03 21:30:50.373017 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:50.372981 4493 patch_prober.go:29] interesting pod/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 03 21:30:50.373204 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:50.373056 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" podUID="01cb721e-c861-49ed-a186-db4cab738a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 03 21:30:50.874521 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:50.874486 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:51.372768 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:51.372732 4493 patch_prober.go:29] interesting pod/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 03 21:30:51.372964 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:51.372800 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" podUID="01cb721e-c861-49ed-a186-db4cab738a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 03 21:30:51.743131 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:30:51.743042 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:30:52.376806 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:52.376773 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:30:52.377717 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:30:52.377698 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:31:21.744537 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:31:21.744502 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:31:51.745798 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:31:51.745758 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:31:58.600568 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:31:58.600530 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88"] Nov 03 21:31:58.708688 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:31:58.708654 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9"] Nov 03 21:32:03.601103 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:03.601037 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88" podUID="ebd6b058-cf2b-444b-97c3-776ccb3d4776" containerName="payload-processing" containerID="cri-o://a09ac51bec4576ee48d867948b99a340e4048bb428efb6604d77dc647b00970f" gracePeriod=25 Nov 03 21:32:03.709430 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:03.709391 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9" podUID="257fe914-4f27-46be-8f87-d4c0376833c6" containerName="payload-pre-processing" containerID="cri-o://fd87ab1be70fa78bcfd3c2110886e4db9645628bc9caaaa0540d8bda9065be37" gracePeriod=25 Nov 03 21:32:03.850989 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:03.850957 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88" Nov 03 21:32:03.866037 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:03.865963 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebd6b058-cf2b-444b-97c3-776ccb3d4776-kube-api-access-wxrgb\" (UniqueName: \"kubernetes.io/projected/ebd6b058-cf2b-444b-97c3-776ccb3d4776-kube-api-access-wxrgb\") pod \"ebd6b058-cf2b-444b-97c3-776ccb3d4776\" (UID: \"ebd6b058-cf2b-444b-97c3-776ccb3d4776\") " Nov 03 21:32:03.866037 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:03.866021 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebd6b058-cf2b-444b-97c3-776ccb3d4776-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd6b058-cf2b-444b-97c3-776ccb3d4776-plugins-config-volume\") pod \"ebd6b058-cf2b-444b-97c3-776ccb3d4776\" (UID: \"ebd6b058-cf2b-444b-97c3-776ccb3d4776\") " Nov 03 21:32:03.866417 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:03.866397 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd6b058-cf2b-444b-97c3-776ccb3d4776-plugins-config-volume" pod "ebd6b058-cf2b-444b-97c3-776ccb3d4776" (UID: "ebd6b058-cf2b-444b-97c3-776ccb3d4776"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:32:03.868218 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:03.868191 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd6b058-cf2b-444b-97c3-776ccb3d4776-kube-api-access-wxrgb" pod "ebd6b058-cf2b-444b-97c3-776ccb3d4776" (UID: "ebd6b058-cf2b-444b-97c3-776ccb3d4776"). InnerVolumeSpecName "kube-api-access-wxrgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:32:03.949756 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:03.949731 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9" Nov 03 21:32:03.966651 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:03.966615 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/257fe914-4f27-46be-8f87-d4c0376833c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/257fe914-4f27-46be-8f87-d4c0376833c6-plugins-config-volume\") pod \"257fe914-4f27-46be-8f87-d4c0376833c6\" (UID: \"257fe914-4f27-46be-8f87-d4c0376833c6\") " Nov 03 21:32:03.966833 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:03.966714 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/257fe914-4f27-46be-8f87-d4c0376833c6-kube-api-access-s7jpp\" (UniqueName: \"kubernetes.io/projected/257fe914-4f27-46be-8f87-d4c0376833c6-kube-api-access-s7jpp\") pod \"257fe914-4f27-46be-8f87-d4c0376833c6\" (UID: \"257fe914-4f27-46be-8f87-d4c0376833c6\") " Nov 03 21:32:03.966989 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:03.966969 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxrgb\" (UniqueName: \"kubernetes.io/projected/ebd6b058-cf2b-444b-97c3-776ccb3d4776-kube-api-access-wxrgb\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:03.967069 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:03.966994 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd6b058-cf2b-444b-97c3-776ccb3d4776-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:03.967069 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:03.967039 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/257fe914-4f27-46be-8f87-d4c0376833c6-plugins-config-volume" pod "257fe914-4f27-46be-8f87-d4c0376833c6" (UID: "257fe914-4f27-46be-8f87-d4c0376833c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:32:03.969074 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:03.969046 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257fe914-4f27-46be-8f87-d4c0376833c6-kube-api-access-s7jpp" pod "257fe914-4f27-46be-8f87-d4c0376833c6" (UID: "257fe914-4f27-46be-8f87-d4c0376833c6"). InnerVolumeSpecName "kube-api-access-s7jpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:32:04.068154 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:04.068122 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7jpp\" (UniqueName: \"kubernetes.io/projected/257fe914-4f27-46be-8f87-d4c0376833c6-kube-api-access-s7jpp\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:04.068154 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:04.068146 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/257fe914-4f27-46be-8f87-d4c0376833c6-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:04.409310 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:04.409278 4493 generic.go:356] "Generic (PLEG): container finished" podID="ebd6b058-cf2b-444b-97c3-776ccb3d4776" containerID="a09ac51bec4576ee48d867948b99a340e4048bb428efb6604d77dc647b00970f" exitCode=0 Nov 03 21:32:04.409495 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:04.409323 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88" Nov 03 21:32:04.409495 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:04.409365 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88" event={"ID":"ebd6b058-cf2b-444b-97c3-776ccb3d4776","Type":"ContainerDied","Data":"a09ac51bec4576ee48d867948b99a340e4048bb428efb6604d77dc647b00970f"} Nov 03 21:32:04.409495 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:04.409463 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88" event={"ID":"ebd6b058-cf2b-444b-97c3-776ccb3d4776","Type":"ContainerDied","Data":"d16ea5f5d1d2b896df9637916a16dbbf77e8036836a123ca944b876cf745e3c9"} Nov 03 21:32:04.409495 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:04.409491 4493 scope.go:122] "RemoveContainer" containerID="a09ac51bec4576ee48d867948b99a340e4048bb428efb6604d77dc647b00970f" Nov 03 21:32:04.410882 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:04.410861 4493 generic.go:356] "Generic (PLEG): container finished" podID="257fe914-4f27-46be-8f87-d4c0376833c6" containerID="fd87ab1be70fa78bcfd3c2110886e4db9645628bc9caaaa0540d8bda9065be37" exitCode=0 Nov 03 21:32:04.410987 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:04.410940 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9" Nov 03 21:32:04.411044 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:04.410944 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9" event={"ID":"257fe914-4f27-46be-8f87-d4c0376833c6","Type":"ContainerDied","Data":"fd87ab1be70fa78bcfd3c2110886e4db9645628bc9caaaa0540d8bda9065be37"} Nov 03 21:32:04.411103 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:04.411059 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9" event={"ID":"257fe914-4f27-46be-8f87-d4c0376833c6","Type":"ContainerDied","Data":"dc079e94a1c15d2326ede398199e6465359ad81d340918de84a86bb33cefcc4f"} Nov 03 21:32:04.418252 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:04.418228 4493 scope.go:122] "RemoveContainer" containerID="a09ac51bec4576ee48d867948b99a340e4048bb428efb6604d77dc647b00970f" Nov 03 21:32:04.418571 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:32:04.418553 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a09ac51bec4576ee48d867948b99a340e4048bb428efb6604d77dc647b00970f\": container with ID starting with a09ac51bec4576ee48d867948b99a340e4048bb428efb6604d77dc647b00970f not found: ID does not exist" containerID="a09ac51bec4576ee48d867948b99a340e4048bb428efb6604d77dc647b00970f" Nov 03 21:32:04.418662 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:04.418578 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a09ac51bec4576ee48d867948b99a340e4048bb428efb6604d77dc647b00970f"} err="failed to get container status \"a09ac51bec4576ee48d867948b99a340e4048bb428efb6604d77dc647b00970f\": rpc error: code = NotFound desc = could not find container \"a09ac51bec4576ee48d867948b99a340e4048bb428efb6604d77dc647b00970f\": container with ID starting with a09ac51bec4576ee48d867948b99a340e4048bb428efb6604d77dc647b00970f not found: ID does not exist" Nov 03 21:32:04.418662 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:04.418595 4493 scope.go:122] "RemoveContainer" containerID="fd87ab1be70fa78bcfd3c2110886e4db9645628bc9caaaa0540d8bda9065be37" Nov 03 21:32:04.426818 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:04.426800 4493 scope.go:122] "RemoveContainer" containerID="fd87ab1be70fa78bcfd3c2110886e4db9645628bc9caaaa0540d8bda9065be37" Nov 03 21:32:04.427081 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:32:04.427055 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd87ab1be70fa78bcfd3c2110886e4db9645628bc9caaaa0540d8bda9065be37\": container with ID starting with fd87ab1be70fa78bcfd3c2110886e4db9645628bc9caaaa0540d8bda9065be37 not found: ID does not exist" containerID="fd87ab1be70fa78bcfd3c2110886e4db9645628bc9caaaa0540d8bda9065be37" Nov 03 21:32:04.427140 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:04.427081 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd87ab1be70fa78bcfd3c2110886e4db9645628bc9caaaa0540d8bda9065be37"} err="failed to get container status \"fd87ab1be70fa78bcfd3c2110886e4db9645628bc9caaaa0540d8bda9065be37\": rpc error: code = NotFound desc = could not find container \"fd87ab1be70fa78bcfd3c2110886e4db9645628bc9caaaa0540d8bda9065be37\": container with ID starting with fd87ab1be70fa78bcfd3c2110886e4db9645628bc9caaaa0540d8bda9065be37 not found: ID does not exist" Nov 03 21:32:04.580485 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:04.580458 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9"] Nov 03 21:32:04.583747 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:04.583726 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6d8154-6bf5fsw8n9"] Nov 03 21:32:04.695862 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:04.695783 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88"] Nov 03 21:32:04.699503 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:04.699479 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6d8154-bb8bf4cc9sgv88"] Nov 03 21:32:06.560892 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:06.560858 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="257fe914-4f27-46be-8f87-d4c0376833c6" path="/var/lib/kubelet/pods/257fe914-4f27-46be-8f87-d4c0376833c6/volumes" Nov 03 21:32:06.561286 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:06.561234 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebd6b058-cf2b-444b-97c3-776ccb3d4776" path="/var/lib/kubelet/pods/ebd6b058-cf2b-444b-97c3-776ccb3d4776/volumes" Nov 03 21:32:10.340489 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:10.340453 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8"] Nov 03 21:32:10.340896 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:10.340662 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" podUID="01cb721e-c861-49ed-a186-db4cab738a85" containerName="istio-proxy" containerID="cri-o://964683f45e79203be1fc8561ada9e887a7dc30ad4a94ab8db1a6206dd0b07784" gracePeriod=30 Nov 03 21:32:15.467775 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.467736 4493 generic.go:356] "Generic (PLEG): container finished" podID="01cb721e-c861-49ed-a186-db4cab738a85" containerID="964683f45e79203be1fc8561ada9e887a7dc30ad4a94ab8db1a6206dd0b07784" exitCode=0 Nov 03 21:32:15.468141 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.467801 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" event={"ID":"01cb721e-c861-49ed-a186-db4cab738a85","Type":"ContainerDied","Data":"964683f45e79203be1fc8561ada9e887a7dc30ad4a94ab8db1a6206dd0b07784"} Nov 03 21:32:15.468141 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.467862 4493 scope.go:122] "RemoveContainer" containerID="2d95b24fc7bc20d6bc38b461f4f4a6cd7475ef6ca8e16d0cbad537f0a06c3fb8" Nov 03 21:32:15.591392 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.591364 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:32:15.681556 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.681483 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-workload-certs\") pod \"01cb721e-c861-49ed-a186-db4cab738a85\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " Nov 03 21:32:15.681556 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.681525 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-istio-data\") pod \"01cb721e-c861-49ed-a186-db4cab738a85\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " Nov 03 21:32:15.681772 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.681567 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-istio-envoy\") pod \"01cb721e-c861-49ed-a186-db4cab738a85\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " Nov 03 21:32:15.681772 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.681596 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01cb721e-c861-49ed-a186-db4cab738a85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01cb721e-c861-49ed-a186-db4cab738a85-istiod-ca-cert\") pod \"01cb721e-c861-49ed-a186-db4cab738a85\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " Nov 03 21:32:15.681772 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.681621 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-credential-socket\") pod \"01cb721e-c861-49ed-a186-db4cab738a85\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " Nov 03 21:32:15.681772 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.681708 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-workload-socket\") pod \"01cb721e-c861-49ed-a186-db4cab738a85\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " Nov 03 21:32:15.681772 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.681749 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01cb721e-c861-49ed-a186-db4cab738a85-kube-api-access-4srkb\" (UniqueName: \"kubernetes.io/projected/01cb721e-c861-49ed-a186-db4cab738a85-kube-api-access-4srkb\") pod \"01cb721e-c861-49ed-a186-db4cab738a85\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " Nov 03 21:32:15.682032 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.681792 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01cb721e-c861-49ed-a186-db4cab738a85-istio-token\" (UniqueName: \"kubernetes.io/projected/01cb721e-c861-49ed-a186-db4cab738a85-istio-token\") pod \"01cb721e-c861-49ed-a186-db4cab738a85\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " Nov 03 21:32:15.682032 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.681841 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01cb721e-c861-49ed-a186-db4cab738a85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01cb721e-c861-49ed-a186-db4cab738a85-istio-podinfo\") pod \"01cb721e-c861-49ed-a186-db4cab738a85\" (UID: \"01cb721e-c861-49ed-a186-db4cab738a85\") " Nov 03 21:32:15.682032 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.681859 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-workload-certs" pod "01cb721e-c861-49ed-a186-db4cab738a85" (UID: "01cb721e-c861-49ed-a186-db4cab738a85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:32:15.682032 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.681877 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-istio-data" pod "01cb721e-c861-49ed-a186-db4cab738a85" (UID: "01cb721e-c861-49ed-a186-db4cab738a85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:32:15.682032 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.682011 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-workload-socket" pod "01cb721e-c861-49ed-a186-db4cab738a85" (UID: "01cb721e-c861-49ed-a186-db4cab738a85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:32:15.682255 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.682051 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-credential-socket" pod "01cb721e-c861-49ed-a186-db4cab738a85" (UID: "01cb721e-c861-49ed-a186-db4cab738a85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:32:15.682255 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.682171 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01cb721e-c861-49ed-a186-db4cab738a85-istiod-ca-cert" pod "01cb721e-c861-49ed-a186-db4cab738a85" (UID: "01cb721e-c861-49ed-a186-db4cab738a85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:32:15.682359 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.682253 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:15.682359 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.682277 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:15.682359 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.682290 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:15.682359 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.682304 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:15.684247 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.684223 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01cb721e-c861-49ed-a186-db4cab738a85-istio-podinfo" pod "01cb721e-c861-49ed-a186-db4cab738a85" (UID: "01cb721e-c861-49ed-a186-db4cab738a85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:32:15.684427 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.684379 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01cb721e-c861-49ed-a186-db4cab738a85-istio-token" pod "01cb721e-c861-49ed-a186-db4cab738a85" (UID: "01cb721e-c861-49ed-a186-db4cab738a85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:32:15.684626 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.684599 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01cb721e-c861-49ed-a186-db4cab738a85-kube-api-access-4srkb" pod "01cb721e-c861-49ed-a186-db4cab738a85" (UID: "01cb721e-c861-49ed-a186-db4cab738a85"). InnerVolumeSpecName "kube-api-access-4srkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:32:15.684626 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.684603 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-istio-envoy" pod "01cb721e-c861-49ed-a186-db4cab738a85" (UID: "01cb721e-c861-49ed-a186-db4cab738a85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:32:15.783815 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.783776 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4srkb\" (UniqueName: \"kubernetes.io/projected/01cb721e-c861-49ed-a186-db4cab738a85-kube-api-access-4srkb\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:15.783815 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.783819 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01cb721e-c861-49ed-a186-db4cab738a85-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:15.784024 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.783834 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01cb721e-c861-49ed-a186-db4cab738a85-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:15.784024 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.783849 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01cb721e-c861-49ed-a186-db4cab738a85-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:15.784024 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:15.783862 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01cb721e-c861-49ed-a186-db4cab738a85-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:16.474071 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:16.474040 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" Nov 03 21:32:16.474558 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:16.474071 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8" event={"ID":"01cb721e-c861-49ed-a186-db4cab738a85","Type":"ContainerDied","Data":"c685f2f8b3ce0d19a359b2d727895dad4268372f3177b328a8d7dbfc29cf55ca"} Nov 03 21:32:16.474558 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:16.474126 4493 scope.go:122] "RemoveContainer" containerID="964683f45e79203be1fc8561ada9e887a7dc30ad4a94ab8db1a6206dd0b07784" Nov 03 21:32:16.694438 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:16.694406 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8"] Nov 03 21:32:16.746090 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:16.746014 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6d8154-openshift-default-8c665c8f-qhfp8"] Nov 03 21:32:18.560484 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:32:18.560451 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01cb721e-c861-49ed-a186-db4cab738a85" path="/var/lib/kubelet/pods/01cb721e-c861-49ed-a186-db4cab738a85/volumes" Nov 03 21:32:21.756046 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:32:21.756007 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:32:51.744516 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:32:51.744471 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:33:05.245360 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:33:05.242604 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-zgpq2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:33:05.245360 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:33:05.242680 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zgpq2" podUID="da127cda-8ef4-43c8-8eb8-2fe89b52ffcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:33:05.282615 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:33:05.282488 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-xd49n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:33:05.282615 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:33:05.282552 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xd49n" podUID="197b41b5-4233-40c8-855a-38b91aedf4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:33:21.744076 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:33:21.744040 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:33:49.295244 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:33:49.295215 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 21:33:49.381993 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:33:49.381963 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 21:33:51.743549 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:33:51.743514 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:34:21.758244 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:34:21.758200 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:34:47.382603 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.382561 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59fffb9f98-4zvg6"] Nov 03 21:34:47.383149 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.383116 4493 cpu_manager.go:414] "containerMap: removing container" podUID="01cb721e-c861-49ed-a186-db4cab738a85" containerName="istio-proxy" Nov 03 21:34:47.383149 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.383147 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01cb721e-c861-49ed-a186-db4cab738a85" containerName="istio-proxy" Nov 03 21:34:47.383397 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.383164 4493 cpu_manager.go:414] "containerMap: removing container" podUID="257fe914-4f27-46be-8f87-d4c0376833c6" containerName="payload-pre-processing" Nov 03 21:34:47.383397 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.383173 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="257fe914-4f27-46be-8f87-d4c0376833c6" containerName="payload-pre-processing" Nov 03 21:34:47.383397 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.383214 4493 cpu_manager.go:414] "containerMap: removing container" podUID="01cb721e-c861-49ed-a186-db4cab738a85" containerName="istio-proxy" Nov 03 21:34:47.383397 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.383222 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01cb721e-c861-49ed-a186-db4cab738a85" containerName="istio-proxy" Nov 03 21:34:47.383397 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.383235 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ebd6b058-cf2b-444b-97c3-776ccb3d4776" containerName="payload-processing" Nov 03 21:34:47.383397 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.383242 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebd6b058-cf2b-444b-97c3-776ccb3d4776" containerName="payload-processing" Nov 03 21:34:47.383397 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.383376 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="257fe914-4f27-46be-8f87-d4c0376833c6" containerName="payload-pre-processing" Nov 03 21:34:47.383397 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.383397 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="01cb721e-c861-49ed-a186-db4cab738a85" containerName="istio-proxy" Nov 03 21:34:47.383729 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.383414 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebd6b058-cf2b-444b-97c3-776ccb3d4776" containerName="payload-processing" Nov 03 21:34:47.383729 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.383429 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="01cb721e-c861-49ed-a186-db4cab738a85" containerName="istio-proxy" Nov 03 21:34:47.387047 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.387026 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59fffb9f98-4zvg6" Nov 03 21:34:47.473996 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.473955 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe60236-83d1-4419-85c2-b99031445c6b-openshift-service-ca\") pod \"maas-api-59fffb9f98-4zvg6\" (UID: \"dfe60236-83d1-4419-85c2-b99031445c6b\") " pod="odh-ai-gateway-infra/maas-api-59fffb9f98-4zvg6" Nov 03 21:34:47.474263 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.474026 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrv8q\" (UniqueName: \"kubernetes.io/projected/dfe60236-83d1-4419-85c2-b99031445c6b-kube-api-access-qrv8q\") pod \"maas-api-59fffb9f98-4zvg6\" (UID: \"dfe60236-83d1-4419-85c2-b99031445c6b\") " pod="odh-ai-gateway-infra/maas-api-59fffb9f98-4zvg6" Nov 03 21:34:47.474263 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.474077 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfe60236-83d1-4419-85c2-b99031445c6b-metrics-certs\") pod \"maas-api-59fffb9f98-4zvg6\" (UID: \"dfe60236-83d1-4419-85c2-b99031445c6b\") " pod="odh-ai-gateway-infra/maas-api-59fffb9f98-4zvg6" Nov 03 21:34:47.474263 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.474126 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfe60236-83d1-4419-85c2-b99031445c6b-maas-api-tls\") pod \"maas-api-59fffb9f98-4zvg6\" (UID: \"dfe60236-83d1-4419-85c2-b99031445c6b\") " pod="odh-ai-gateway-infra/maas-api-59fffb9f98-4zvg6" Nov 03 21:34:47.496418 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.496379 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59fffb9f98-4zvg6"] Nov 03 21:34:47.574729 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.574693 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfe60236-83d1-4419-85c2-b99031445c6b-metrics-certs\") pod \"maas-api-59fffb9f98-4zvg6\" (UID: \"dfe60236-83d1-4419-85c2-b99031445c6b\") " pod="odh-ai-gateway-infra/maas-api-59fffb9f98-4zvg6" Nov 03 21:34:47.574933 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.574740 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfe60236-83d1-4419-85c2-b99031445c6b-maas-api-tls\") pod \"maas-api-59fffb9f98-4zvg6\" (UID: \"dfe60236-83d1-4419-85c2-b99031445c6b\") " pod="odh-ai-gateway-infra/maas-api-59fffb9f98-4zvg6" Nov 03 21:34:47.574933 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.574794 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe60236-83d1-4419-85c2-b99031445c6b-openshift-service-ca\") pod \"maas-api-59fffb9f98-4zvg6\" (UID: \"dfe60236-83d1-4419-85c2-b99031445c6b\") " pod="odh-ai-gateway-infra/maas-api-59fffb9f98-4zvg6" Nov 03 21:34:47.574933 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.574837 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrv8q\" (UniqueName: \"kubernetes.io/projected/dfe60236-83d1-4419-85c2-b99031445c6b-kube-api-access-qrv8q\") pod \"maas-api-59fffb9f98-4zvg6\" (UID: \"dfe60236-83d1-4419-85c2-b99031445c6b\") " pod="odh-ai-gateway-infra/maas-api-59fffb9f98-4zvg6" Nov 03 21:34:47.575552 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.575521 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe60236-83d1-4419-85c2-b99031445c6b-openshift-service-ca\") pod \"maas-api-59fffb9f98-4zvg6\" (UID: \"dfe60236-83d1-4419-85c2-b99031445c6b\") " pod="odh-ai-gateway-infra/maas-api-59fffb9f98-4zvg6" Nov 03 21:34:47.577633 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.577607 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfe60236-83d1-4419-85c2-b99031445c6b-metrics-certs\") pod \"maas-api-59fffb9f98-4zvg6\" (UID: \"dfe60236-83d1-4419-85c2-b99031445c6b\") " pod="odh-ai-gateway-infra/maas-api-59fffb9f98-4zvg6" Nov 03 21:34:47.577762 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.577741 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfe60236-83d1-4419-85c2-b99031445c6b-maas-api-tls\") pod \"maas-api-59fffb9f98-4zvg6\" (UID: \"dfe60236-83d1-4419-85c2-b99031445c6b\") " pod="odh-ai-gateway-infra/maas-api-59fffb9f98-4zvg6" Nov 03 21:34:47.633455 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.633384 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrv8q\" (UniqueName: \"kubernetes.io/projected/dfe60236-83d1-4419-85c2-b99031445c6b-kube-api-access-qrv8q\") pod \"maas-api-59fffb9f98-4zvg6\" (UID: \"dfe60236-83d1-4419-85c2-b99031445c6b\") " pod="odh-ai-gateway-infra/maas-api-59fffb9f98-4zvg6" Nov 03 21:34:47.702195 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:47.702166 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59fffb9f98-4zvg6" Nov 03 21:34:48.074435 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:48.074405 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59fffb9f98-4zvg6"] Nov 03 21:34:48.075466 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:34:48.075441 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe60236_83d1_4419_85c2_b99031445c6b.slice/crio-e9230e0e462df22cd0c206caf61853c09d693f5b837d93e89530bcfffda396e7 WatchSource:0}: Error finding container e9230e0e462df22cd0c206caf61853c09d693f5b837d93e89530bcfffda396e7: Status 404 returned error can't find the container with id e9230e0e462df22cd0c206caf61853c09d693f5b837d93e89530bcfffda396e7 Nov 03 21:34:48.076777 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:48.076761 4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:34:48.328036 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:48.327960 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59fffb9f98-4zvg6" event={"ID":"dfe60236-83d1-4419-85c2-b99031445c6b","Type":"ContainerStarted","Data":"e9230e0e462df22cd0c206caf61853c09d693f5b837d93e89530bcfffda396e7"} Nov 03 21:34:49.334262 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:49.334229 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59fffb9f98-4zvg6" event={"ID":"dfe60236-83d1-4419-85c2-b99031445c6b","Type":"ContainerStarted","Data":"44840c0694c67608a680ba5e2be56abaa85fa7a002510c9cd477d6c93a21b6dc"} Nov 03 21:34:49.334972 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:49.334304 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59fffb9f98-4zvg6" Nov 03 21:34:49.452287 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:49.452228 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59fffb9f98-4zvg6" podStartSLOduration=1.573681401 podStartE2EDuration="2.45221503s" totalImagesPullingTime="878.533629ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:49.451181818 +0000 UTC m=+3961.009172317" watchObservedRunningTime="2026-11-03 21:34:49.45221503 +0000 UTC m=+3961.010205539" Nov 03 21:34:51.743325 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:34:51.743285 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:34:54.495383 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.495342 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj"] Nov 03 21:34:54.499069 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.499052 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.552160 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.552120 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-istio-envoy\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.552344 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.552180 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-credential-socket\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.552344 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.552235 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-istio-data\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.552426 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.552391 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-workload-certs\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.552461 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.552420 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4990b9-156a-4777-a11c-7b470d725903-istio-podinfo\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.552461 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.552446 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4990b9-156a-4777-a11c-7b470d725903-istiod-ca-cert\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.552563 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.552538 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f4990b9-156a-4777-a11c-7b470d725903-istio-token\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.552642 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.552627 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-workload-socket\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.552728 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.552710 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwhtf\" (UniqueName: \"kubernetes.io/projected/7f4990b9-156a-4777-a11c-7b470d725903-kube-api-access-pwhtf\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.554903 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.554885 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-62df72ff-openshift-default-dockercfg-4cq7w\"" Nov 03 21:34:54.608119 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.608082 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj"] Nov 03 21:34:54.653858 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.653816 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwhtf\" (UniqueName: \"kubernetes.io/projected/7f4990b9-156a-4777-a11c-7b470d725903-kube-api-access-pwhtf\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.654068 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.653901 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-istio-envoy\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.654068 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.653940 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-credential-socket\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.654068 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.653963 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-istio-data\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.654068 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.653993 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-workload-certs\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.654068 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.654020 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4990b9-156a-4777-a11c-7b470d725903-istio-podinfo\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.654068 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.654048 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4990b9-156a-4777-a11c-7b470d725903-istiod-ca-cert\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.654421 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.654075 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f4990b9-156a-4777-a11c-7b470d725903-istio-token\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.654421 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.654125 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-workload-socket\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.654637 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.654615 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-workload-socket\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.655502 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.655473 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-workload-certs\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.655687 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.655662 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-credential-socket\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.656068 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.656048 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4990b9-156a-4777-a11c-7b470d725903-istiod-ca-cert\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.656192 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.656145 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-istio-data\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.658299 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.658272 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-istio-envoy\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.659399 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.659379 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4990b9-156a-4777-a11c-7b470d725903-istio-podinfo\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.718551 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.718523 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f4990b9-156a-4777-a11c-7b470d725903-istio-token\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.718834 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.718810 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwhtf\" (UniqueName: \"kubernetes.io/projected/7f4990b9-156a-4777-a11c-7b470d725903-kube-api-access-pwhtf\") pod \"e2e-ait-62df72ff-openshift-default-dcb74987-574xj\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:54.812099 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:54.812067 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:55.250447 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:55.250424 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj"] Nov 03 21:34:55.251949 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:34:55.251925 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f4990b9_156a_4777_a11c_7b470d725903.slice/crio-70a3d540adba0089b9e4577135e8868bb0df6de6b7c37093d94e612df599478a WatchSource:0}: Error finding container 70a3d540adba0089b9e4577135e8868bb0df6de6b7c37093d94e612df599478a: Status 404 returned error can't find the container with id 70a3d540adba0089b9e4577135e8868bb0df6de6b7c37093d94e612df599478a Nov 03 21:34:55.254131 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:55.254100 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:34:55.254208 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:55.254167 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:34:55.254208 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:55.254195 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:34:55.345534 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:55.345395 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59fffb9f98-4zvg6" Nov 03 21:34:55.364736 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:55.364703 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" event={"ID":"7f4990b9-156a-4777-a11c-7b470d725903","Type":"ContainerStarted","Data":"5f5116c05e288ff743426ac79cadb1d2f11f6e6fec6cc008f7d0147fa1d042fa"} Nov 03 21:34:55.364901 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:55.364762 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" event={"ID":"7f4990b9-156a-4777-a11c-7b470d725903","Type":"ContainerStarted","Data":"70a3d540adba0089b9e4577135e8868bb0df6de6b7c37093d94e612df599478a"} Nov 03 21:34:55.512949 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:55.512870 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn"] Nov 03 21:34:55.513368 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:55.513117 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn" podUID="feae879c-5eba-47e8-bb99-c6bcd5939ab0" containerName="maas-api" containerID="cri-o://8e3642a58aef8d98af5d1a2e305434b12ace3e82341f0c1cba93626bd624cda4" gracePeriod=30 Nov 03 21:34:55.593413 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:55.593351 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" podStartSLOduration=1.593315984 podStartE2EDuration="1.593315984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:55.592479935 +0000 UTC m=+3967.150470446" watchObservedRunningTime="2026-11-03 21:34:55.593315984 +0000 UTC m=+3967.151306482" Nov 03 21:34:55.812668 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:55.812616 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:56.762193 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:56.762170 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn" Nov 03 21:34:56.818248 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:56.818223 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:56.877788 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:56.877722 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feae879c-5eba-47e8-bb99-c6bcd5939ab0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feae879c-5eba-47e8-bb99-c6bcd5939ab0-openshift-service-ca\") pod \"feae879c-5eba-47e8-bb99-c6bcd5939ab0\" (UID: \"feae879c-5eba-47e8-bb99-c6bcd5939ab0\") " Nov 03 21:34:56.877936 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:56.877814 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feae879c-5eba-47e8-bb99-c6bcd5939ab0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/feae879c-5eba-47e8-bb99-c6bcd5939ab0-metrics-certs\") pod \"feae879c-5eba-47e8-bb99-c6bcd5939ab0\" (UID: \"feae879c-5eba-47e8-bb99-c6bcd5939ab0\") " Nov 03 21:34:56.877936 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:56.877880 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feae879c-5eba-47e8-bb99-c6bcd5939ab0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feae879c-5eba-47e8-bb99-c6bcd5939ab0-maas-api-tls\") pod \"feae879c-5eba-47e8-bb99-c6bcd5939ab0\" (UID: \"feae879c-5eba-47e8-bb99-c6bcd5939ab0\") " Nov 03 21:34:56.878052 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:56.877948 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feae879c-5eba-47e8-bb99-c6bcd5939ab0-kube-api-access-pjpv5\" (UniqueName: \"kubernetes.io/projected/feae879c-5eba-47e8-bb99-c6bcd5939ab0-kube-api-access-pjpv5\") pod \"feae879c-5eba-47e8-bb99-c6bcd5939ab0\" (UID: \"feae879c-5eba-47e8-bb99-c6bcd5939ab0\") " Nov 03 21:34:56.878148 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:56.878118 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feae879c-5eba-47e8-bb99-c6bcd5939ab0-openshift-service-ca" pod "feae879c-5eba-47e8-bb99-c6bcd5939ab0" (UID: "feae879c-5eba-47e8-bb99-c6bcd5939ab0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:34:56.878532 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:56.878318 4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feae879c-5eba-47e8-bb99-c6bcd5939ab0-openshift-service-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:34:56.882518 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:56.882459 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feae879c-5eba-47e8-bb99-c6bcd5939ab0-maas-api-tls" pod "feae879c-5eba-47e8-bb99-c6bcd5939ab0" (UID: "feae879c-5eba-47e8-bb99-c6bcd5939ab0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:34:56.882518 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:56.882477 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feae879c-5eba-47e8-bb99-c6bcd5939ab0-kube-api-access-pjpv5" pod "feae879c-5eba-47e8-bb99-c6bcd5939ab0" (UID: "feae879c-5eba-47e8-bb99-c6bcd5939ab0"). InnerVolumeSpecName "kube-api-access-pjpv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:34:56.882518 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:56.882479 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feae879c-5eba-47e8-bb99-c6bcd5939ab0-metrics-certs" pod "feae879c-5eba-47e8-bb99-c6bcd5939ab0" (UID: "feae879c-5eba-47e8-bb99-c6bcd5939ab0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:34:56.979588 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:56.979559 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feae879c-5eba-47e8-bb99-c6bcd5939ab0-maas-api-tls\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:34:56.979588 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:56.979587 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjpv5\" (UniqueName: \"kubernetes.io/projected/feae879c-5eba-47e8-bb99-c6bcd5939ab0-kube-api-access-pjpv5\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:34:56.979793 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:56.979603 4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feae879c-5eba-47e8-bb99-c6bcd5939ab0-metrics-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:34:57.375646 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:57.375612 4493 generic.go:356] "Generic (PLEG): container finished" podID="feae879c-5eba-47e8-bb99-c6bcd5939ab0" containerID="8e3642a58aef8d98af5d1a2e305434b12ace3e82341f0c1cba93626bd624cda4" exitCode=0 Nov 03 21:34:57.375826 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:57.375669 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn" Nov 03 21:34:57.375826 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:57.375692 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn" event={"ID":"feae879c-5eba-47e8-bb99-c6bcd5939ab0","Type":"ContainerDied","Data":"8e3642a58aef8d98af5d1a2e305434b12ace3e82341f0c1cba93626bd624cda4"} Nov 03 21:34:57.375826 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:57.375766 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn" event={"ID":"feae879c-5eba-47e8-bb99-c6bcd5939ab0","Type":"ContainerDied","Data":"3cc0b42b4b0ebfc18d8e9cfb72c9639c7f1ceee83aa374ba307a94638ed4469d"} Nov 03 21:34:57.375826 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:57.375787 4493 scope.go:122] "RemoveContainer" containerID="8e3642a58aef8d98af5d1a2e305434b12ace3e82341f0c1cba93626bd624cda4" Nov 03 21:34:57.376198 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:57.376174 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:57.377381 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:57.377365 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:34:57.385680 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:57.385664 4493 scope.go:122] "RemoveContainer" containerID="8e3642a58aef8d98af5d1a2e305434b12ace3e82341f0c1cba93626bd624cda4" Nov 03 21:34:57.385943 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:34:57.385924 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e3642a58aef8d98af5d1a2e305434b12ace3e82341f0c1cba93626bd624cda4\": container with ID starting with 8e3642a58aef8d98af5d1a2e305434b12ace3e82341f0c1cba93626bd624cda4 not found: ID does not exist" containerID="8e3642a58aef8d98af5d1a2e305434b12ace3e82341f0c1cba93626bd624cda4" Nov 03 21:34:57.386013 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:57.385950 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e3642a58aef8d98af5d1a2e305434b12ace3e82341f0c1cba93626bd624cda4"} err="failed to get container status \"8e3642a58aef8d98af5d1a2e305434b12ace3e82341f0c1cba93626bd624cda4\": rpc error: code = NotFound desc = could not find container \"8e3642a58aef8d98af5d1a2e305434b12ace3e82341f0c1cba93626bd624cda4\": container with ID starting with 8e3642a58aef8d98af5d1a2e305434b12ace3e82341f0c1cba93626bd624cda4 not found: ID does not exist" Nov 03 21:34:57.543518 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:57.543484 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn"] Nov 03 21:34:57.545874 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:57.545854 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ff6ff6f8b-2blhn"] Nov 03 21:34:58.559785 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:34:58.559754 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feae879c-5eba-47e8-bb99-c6bcd5939ab0" path="/var/lib/kubelet/pods/feae879c-5eba-47e8-bb99-c6bcd5939ab0/volumes" Nov 03 21:35:01.300608 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.300578 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8"] Nov 03 21:35:01.301002 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.300969 4493 cpu_manager.go:414] "containerMap: removing container" podUID="feae879c-5eba-47e8-bb99-c6bcd5939ab0" containerName="maas-api" Nov 03 21:35:01.301002 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.300980 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feae879c-5eba-47e8-bb99-c6bcd5939ab0" containerName="maas-api" Nov 03 21:35:01.301075 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.301040 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="feae879c-5eba-47e8-bb99-c6bcd5939ab0" containerName="maas-api" Nov 03 21:35:01.305525 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.305509 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8" Nov 03 21:35:01.360236 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.360188 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8"] Nov 03 21:35:01.409628 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.409596 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-62df72ff\"" Nov 03 21:35:01.409821 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.409705 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-62df72ff-dockercfg-9rmh9\"" Nov 03 21:35:01.414261 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.414236 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-62df72ff-64d485b449-25tsr"] Nov 03 21:35:01.417825 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.417800 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-62df72ff-64d485b449-25tsr" Nov 03 21:35:01.421287 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.421268 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6faa2a03-f985-4d4d-a458-58ae4afe04aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8\" (UID: \"6faa2a03-f985-4d4d-a458-58ae4afe04aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8" Nov 03 21:35:01.421432 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.421345 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxxjj\" (UniqueName: \"kubernetes.io/projected/6faa2a03-f985-4d4d-a458-58ae4afe04aa-kube-api-access-mxxjj\") pod \"payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8\" (UID: \"6faa2a03-f985-4d4d-a458-58ae4afe04aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8" Nov 03 21:35:01.475494 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.470984 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-62df72ff-64d485b449-25tsr"] Nov 03 21:35:01.522327 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.522290 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6faa2a03-f985-4d4d-a458-58ae4afe04aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8\" (UID: \"6faa2a03-f985-4d4d-a458-58ae4afe04aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8" Nov 03 21:35:01.522327 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.522355 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94hqd\" (UniqueName: \"kubernetes.io/projected/ed257ba2-2bf9-4a48-8254-2ec8a11bb14a-kube-api-access-94hqd\") pod \"payload-processing-e2e-ait-62df72ff-64d485b449-25tsr\" (UID: \"ed257ba2-2bf9-4a48-8254-2ec8a11bb14a\") " pod="openshift-ingress/payload-processing-e2e-ait-62df72ff-64d485b449-25tsr" Nov 03 21:35:01.522591 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.522416 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed257ba2-2bf9-4a48-8254-2ec8a11bb14a-plugins-config-volume\") pod \"payload-processing-e2e-ait-62df72ff-64d485b449-25tsr\" (UID: \"ed257ba2-2bf9-4a48-8254-2ec8a11bb14a\") " pod="openshift-ingress/payload-processing-e2e-ait-62df72ff-64d485b449-25tsr" Nov 03 21:35:01.522591 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.522568 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxxjj\" (UniqueName: \"kubernetes.io/projected/6faa2a03-f985-4d4d-a458-58ae4afe04aa-kube-api-access-mxxjj\") pod \"payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8\" (UID: \"6faa2a03-f985-4d4d-a458-58ae4afe04aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8" Nov 03 21:35:01.523110 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.523090 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6faa2a03-f985-4d4d-a458-58ae4afe04aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8\" (UID: \"6faa2a03-f985-4d4d-a458-58ae4afe04aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8" Nov 03 21:35:01.581950 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.581868 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxxjj\" (UniqueName: \"kubernetes.io/projected/6faa2a03-f985-4d4d-a458-58ae4afe04aa-kube-api-access-mxxjj\") pod \"payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8\" (UID: \"6faa2a03-f985-4d4d-a458-58ae4afe04aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8" Nov 03 21:35:01.617636 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.617602 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8" Nov 03 21:35:01.624063 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.624039 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94hqd\" (UniqueName: \"kubernetes.io/projected/ed257ba2-2bf9-4a48-8254-2ec8a11bb14a-kube-api-access-94hqd\") pod \"payload-processing-e2e-ait-62df72ff-64d485b449-25tsr\" (UID: \"ed257ba2-2bf9-4a48-8254-2ec8a11bb14a\") " pod="openshift-ingress/payload-processing-e2e-ait-62df72ff-64d485b449-25tsr" Nov 03 21:35:01.624267 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.624242 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed257ba2-2bf9-4a48-8254-2ec8a11bb14a-plugins-config-volume\") pod \"payload-processing-e2e-ait-62df72ff-64d485b449-25tsr\" (UID: \"ed257ba2-2bf9-4a48-8254-2ec8a11bb14a\") " pod="openshift-ingress/payload-processing-e2e-ait-62df72ff-64d485b449-25tsr" Nov 03 21:35:01.624934 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.624912 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed257ba2-2bf9-4a48-8254-2ec8a11bb14a-plugins-config-volume\") pod \"payload-processing-e2e-ait-62df72ff-64d485b449-25tsr\" (UID: \"ed257ba2-2bf9-4a48-8254-2ec8a11bb14a\") " pod="openshift-ingress/payload-processing-e2e-ait-62df72ff-64d485b449-25tsr" Nov 03 21:35:01.684388 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.684319 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94hqd\" (UniqueName: \"kubernetes.io/projected/ed257ba2-2bf9-4a48-8254-2ec8a11bb14a-kube-api-access-94hqd\") pod \"payload-processing-e2e-ait-62df72ff-64d485b449-25tsr\" (UID: \"ed257ba2-2bf9-4a48-8254-2ec8a11bb14a\") " pod="openshift-ingress/payload-processing-e2e-ait-62df72ff-64d485b449-25tsr" Nov 03 21:35:01.710106 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.710074 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5"] Nov 03 21:35:01.714864 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.714827 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5" Nov 03 21:35:01.731266 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.731247 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-62df72ff-64d485b449-25tsr" Nov 03 21:35:01.768294 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.768260 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5"] Nov 03 21:35:01.768753 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.768734 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-62df72ff-serving-cert\"" Nov 03 21:35:01.826230 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.826138 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/202b5530-4a86-4bbe-a6f7-b41af4da46ca-maas-api-tls\") pod \"maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5\" (UID: \"202b5530-4a86-4bbe-a6f7-b41af4da46ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5" Nov 03 21:35:01.826230 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.826181 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/202b5530-4a86-4bbe-a6f7-b41af4da46ca-openshift-service-ca\") pod \"maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5\" (UID: \"202b5530-4a86-4bbe-a6f7-b41af4da46ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5" Nov 03 21:35:01.826504 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.826411 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnflt\" (UniqueName: \"kubernetes.io/projected/202b5530-4a86-4bbe-a6f7-b41af4da46ca-kube-api-access-qnflt\") pod \"maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5\" (UID: \"202b5530-4a86-4bbe-a6f7-b41af4da46ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5" Nov 03 21:35:01.826504 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.826461 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/202b5530-4a86-4bbe-a6f7-b41af4da46ca-metrics-certs\") pod \"maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5\" (UID: \"202b5530-4a86-4bbe-a6f7-b41af4da46ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5" Nov 03 21:35:01.927299 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.927207 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnflt\" (UniqueName: \"kubernetes.io/projected/202b5530-4a86-4bbe-a6f7-b41af4da46ca-kube-api-access-qnflt\") pod \"maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5\" (UID: \"202b5530-4a86-4bbe-a6f7-b41af4da46ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5" Nov 03 21:35:01.927299 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.927258 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/202b5530-4a86-4bbe-a6f7-b41af4da46ca-metrics-certs\") pod \"maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5\" (UID: \"202b5530-4a86-4bbe-a6f7-b41af4da46ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5" Nov 03 21:35:01.927573 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.927357 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/202b5530-4a86-4bbe-a6f7-b41af4da46ca-maas-api-tls\") pod \"maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5\" (UID: \"202b5530-4a86-4bbe-a6f7-b41af4da46ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5" Nov 03 21:35:01.927573 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:35:01.927440 4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-serving-cert: secret "maas-api-e2e-ait-62df72ff-serving-cert" not found Nov 03 21:35:01.927573 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.927504 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/202b5530-4a86-4bbe-a6f7-b41af4da46ca-openshift-service-ca\") pod \"maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5\" (UID: \"202b5530-4a86-4bbe-a6f7-b41af4da46ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5" Nov 03 21:35:01.927714 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:35:01.927634 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/202b5530-4a86-4bbe-a6f7-b41af4da46ca-maas-api-tls podName:202b5530-4a86-4bbe-a6f7-b41af4da46ca nodeName:}" failed. No retries permitted until 2026-11-03 21:35:02.427609575 +0000 UTC m=+3973.985600052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/202b5530-4a86-4bbe-a6f7-b41af4da46ca-maas-api-tls") pod "maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5" (UID: "202b5530-4a86-4bbe-a6f7-b41af4da46ca") : secret "maas-api-e2e-ait-62df72ff-serving-cert" not found Nov 03 21:35:01.928155 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.928088 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/202b5530-4a86-4bbe-a6f7-b41af4da46ca-openshift-service-ca\") pod \"maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5\" (UID: \"202b5530-4a86-4bbe-a6f7-b41af4da46ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5" Nov 03 21:35:01.929975 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.929953 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/202b5530-4a86-4bbe-a6f7-b41af4da46ca-metrics-certs\") pod \"maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5\" (UID: \"202b5530-4a86-4bbe-a6f7-b41af4da46ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5" Nov 03 21:35:01.987266 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.987234 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnflt\" (UniqueName: \"kubernetes.io/projected/202b5530-4a86-4bbe-a6f7-b41af4da46ca-kube-api-access-qnflt\") pod \"maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5\" (UID: \"202b5530-4a86-4bbe-a6f7-b41af4da46ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5" Nov 03 21:35:01.990600 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:01.990574 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8"] Nov 03 21:35:01.991871 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:35:01.991844 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6faa2a03_f985_4d4d_a458_58ae4afe04aa.slice/crio-d62b6aad9dec1fa311309fdb4d139cb6ff3410bdcfb8278edfdfc16148bf1e3e WatchSource:0}: Error finding container d62b6aad9dec1fa311309fdb4d139cb6ff3410bdcfb8278edfdfc16148bf1e3e: Status 404 returned error can't find the container with id d62b6aad9dec1fa311309fdb4d139cb6ff3410bdcfb8278edfdfc16148bf1e3e Nov 03 21:35:02.105519 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:02.105482 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-62df72ff-64d485b449-25tsr"] Nov 03 21:35:02.109208 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:35:02.109175 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded257ba2_2bf9_4a48_8254_2ec8a11bb14a.slice/crio-e4698e26a468d042f21d7aafb51dc4253ca974a03e948ec740898e23b22c58f2 WatchSource:0}: Error finding container e4698e26a468d042f21d7aafb51dc4253ca974a03e948ec740898e23b22c58f2: Status 404 returned error can't find the container with id e4698e26a468d042f21d7aafb51dc4253ca974a03e948ec740898e23b22c58f2 Nov 03 21:35:02.402511 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:02.402479 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8" event={"ID":"6faa2a03-f985-4d4d-a458-58ae4afe04aa","Type":"ContainerStarted","Data":"ea21bbc1505849a6c87cee9a78b57dad5e6abd6f1d41167f9b558e036285f010"} Nov 03 21:35:02.402966 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:02.402536 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8" event={"ID":"6faa2a03-f985-4d4d-a458-58ae4afe04aa","Type":"ContainerStarted","Data":"d62b6aad9dec1fa311309fdb4d139cb6ff3410bdcfb8278edfdfc16148bf1e3e"} Nov 03 21:35:02.402966 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:02.402567 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8" Nov 03 21:35:02.404154 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:02.404128 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-62df72ff-64d485b449-25tsr" event={"ID":"ed257ba2-2bf9-4a48-8254-2ec8a11bb14a","Type":"ContainerStarted","Data":"0e2619c467f0411e941a6a912389786f0929d6d36f5e788504d0e3f99853a2b0"} Nov 03 21:35:02.404284 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:02.404170 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-62df72ff-64d485b449-25tsr" event={"ID":"ed257ba2-2bf9-4a48-8254-2ec8a11bb14a","Type":"ContainerStarted","Data":"e4698e26a468d042f21d7aafb51dc4253ca974a03e948ec740898e23b22c58f2"} Nov 03 21:35:02.404284 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:02.404192 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-62df72ff-64d485b449-25tsr" Nov 03 21:35:02.432917 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:02.432879 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/202b5530-4a86-4bbe-a6f7-b41af4da46ca-maas-api-tls\") pod \"maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5\" (UID: \"202b5530-4a86-4bbe-a6f7-b41af4da46ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5" Nov 03 21:35:02.435513 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:02.435487 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/202b5530-4a86-4bbe-a6f7-b41af4da46ca-maas-api-tls\") pod \"maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5\" (UID: \"202b5530-4a86-4bbe-a6f7-b41af4da46ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5" Nov 03 21:35:02.520128 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:02.520079 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8" podStartSLOduration=1.520065873 podStartE2EDuration="1.520065873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:02.519727474 +0000 UTC m=+3974.077717998" watchObservedRunningTime="2026-11-03 21:35:02.520065873 +0000 UTC m=+3974.078056372" Nov 03 21:35:02.627798 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:02.627754 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5" Nov 03 21:35:02.640981 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:02.639064 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-62df72ff-64d485b449-25tsr" podStartSLOduration=1.639045944 podStartE2EDuration="1.639045944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:02.635578484 +0000 UTC m=+3974.193568984" watchObservedRunningTime="2026-11-03 21:35:02.639045944 +0000 UTC m=+3974.197036445" Nov 03 21:35:03.012053 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:03.012024 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5"] Nov 03 21:35:03.012398 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:35:03.012377 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202b5530_4a86_4bbe_a6f7_b41af4da46ca.slice/crio-e4527888ca9403319b830c97c77757c365ab9e185dabdd7d5052b951c81456ab WatchSource:0}: Error finding container e4527888ca9403319b830c97c77757c365ab9e185dabdd7d5052b951c81456ab: Status 404 returned error can't find the container with id e4527888ca9403319b830c97c77757c365ab9e185dabdd7d5052b951c81456ab Nov 03 21:35:03.409950 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:03.409918 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5" event={"ID":"202b5530-4a86-4bbe-a6f7-b41af4da46ca","Type":"ContainerStarted","Data":"e4527888ca9403319b830c97c77757c365ab9e185dabdd7d5052b951c81456ab"} Nov 03 21:35:04.417383 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:04.417321 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5" event={"ID":"202b5530-4a86-4bbe-a6f7-b41af4da46ca","Type":"ContainerStarted","Data":"1dbb49d913192f5bfd21944780ed96c7e57915c04e5132fd13d42e355c9633e6"} Nov 03 21:35:04.417805 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:04.417550 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5" Nov 03 21:35:04.535302 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:04.535245 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5" podStartSLOduration=2.6778737 podStartE2EDuration="3.535232642s" totalImagesPullingTime="857.358942ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:04.534351698 +0000 UTC m=+3976.092342217" watchObservedRunningTime="2026-11-03 21:35:04.535232642 +0000 UTC m=+3976.093223205" Nov 03 21:35:10.427829 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:10.427800 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5" Nov 03 21:35:13.413747 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:13.413719 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-62df72ff-64d485b449-25tsr" Nov 03 21:35:13.414147 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:13.413793 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8" Nov 03 21:35:19.499744 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:19.499668 4493 generic.go:356] "Generic (PLEG): container finished" podID="7f4990b9-156a-4777-a11c-7b470d725903" containerID="5f5116c05e288ff743426ac79cadb1d2f11f6e6fec6cc008f7d0147fa1d042fa" exitCode=0 Nov 03 21:35:19.500135 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:19.499740 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" event={"ID":"7f4990b9-156a-4777-a11c-7b470d725903","Type":"ContainerDied","Data":"5f5116c05e288ff743426ac79cadb1d2f11f6e6fec6cc008f7d0147fa1d042fa"} Nov 03 21:35:19.500135 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:19.500115 4493 scope.go:122] "RemoveContainer" containerID="5f5116c05e288ff743426ac79cadb1d2f11f6e6fec6cc008f7d0147fa1d042fa" Nov 03 21:35:19.501390 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:19.501364 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:35:19.501475 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:19.501439 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:35:19.501513 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:19.501475 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:35:20.511602 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:20.511554 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" event={"ID":"7f4990b9-156a-4777-a11c-7b470d725903","Type":"ContainerStarted","Data":"490e1e545d7f9c806ae3f4d02c78d13f33c54c8a0aecde7424ba61068f6a3bb7"} Nov 03 21:35:20.813210 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:20.813170 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:35:20.813487 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:20.813462 4493 patch_prober.go:29] interesting pod/e2e-ait-62df72ff-openshift-default-dcb74987-574xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 21:35:20.813543 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:20.813525 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" podUID="7f4990b9-156a-4777-a11c-7b470d725903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 21:35:21.743650 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:35:21.743602 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:35:21.813433 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:21.813400 4493 patch_prober.go:29] interesting pod/e2e-ait-62df72ff-openshift-default-dcb74987-574xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 21:35:21.813610 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:21.813457 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" podUID="7f4990b9-156a-4777-a11c-7b470d725903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 21:35:22.812854 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:22.812817 4493 patch_prober.go:29] interesting pod/e2e-ait-62df72ff-openshift-default-dcb74987-574xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 21:35:22.813280 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:22.812870 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" podUID="7f4990b9-156a-4777-a11c-7b470d725903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 21:35:23.813025 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:23.812962 4493 patch_prober.go:29] interesting pod/e2e-ait-62df72ff-openshift-default-dcb74987-574xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 21:35:23.813574 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:23.813045 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" podUID="7f4990b9-156a-4777-a11c-7b470d725903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 21:35:24.812860 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:24.812823 4493 patch_prober.go:29] interesting pod/e2e-ait-62df72ff-openshift-default-dcb74987-574xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 21:35:24.813102 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:24.812888 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" podUID="7f4990b9-156a-4777-a11c-7b470d725903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 21:35:25.817366 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:25.817317 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:35:25.817847 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:25.817717 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:35:25.818520 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:25.818505 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:35:50.320849 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:50.320816 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-62df72ff-64d485b449-25tsr"] Nov 03 21:35:50.428814 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:50.428780 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8"] Nov 03 21:35:51.743115 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:35:51.743080 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:35:55.322111 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.322052 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-62df72ff-64d485b449-25tsr" podUID="ed257ba2-2bf9-4a48-8254-2ec8a11bb14a" containerName="payload-processing" containerID="cri-o://0e2619c467f0411e941a6a912389786f0929d6d36f5e788504d0e3f99853a2b0" gracePeriod=25 Nov 03 21:35:55.429214 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.429176 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8" podUID="6faa2a03-f985-4d4d-a458-58ae4afe04aa" containerName="payload-pre-processing" containerID="cri-o://ea21bbc1505849a6c87cee9a78b57dad5e6abd6f1d41167f9b558e036285f010" gracePeriod=25 Nov 03 21:35:55.585263 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.585195 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-62df72ff-64d485b449-25tsr" Nov 03 21:35:55.688782 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.688757 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8" Nov 03 21:35:55.694452 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.694422 4493 generic.go:356] "Generic (PLEG): container finished" podID="6faa2a03-f985-4d4d-a458-58ae4afe04aa" containerID="ea21bbc1505849a6c87cee9a78b57dad5e6abd6f1d41167f9b558e036285f010" exitCode=0 Nov 03 21:35:55.694623 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.694478 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8" Nov 03 21:35:55.694623 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.694506 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8" event={"ID":"6faa2a03-f985-4d4d-a458-58ae4afe04aa","Type":"ContainerDied","Data":"ea21bbc1505849a6c87cee9a78b57dad5e6abd6f1d41167f9b558e036285f010"} Nov 03 21:35:55.694623 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.694565 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8" event={"ID":"6faa2a03-f985-4d4d-a458-58ae4afe04aa","Type":"ContainerDied","Data":"d62b6aad9dec1fa311309fdb4d139cb6ff3410bdcfb8278edfdfc16148bf1e3e"} Nov 03 21:35:55.694623 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.694593 4493 scope.go:122] "RemoveContainer" containerID="ea21bbc1505849a6c87cee9a78b57dad5e6abd6f1d41167f9b558e036285f010" Nov 03 21:35:55.695877 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.695859 4493 generic.go:356] "Generic (PLEG): container finished" podID="ed257ba2-2bf9-4a48-8254-2ec8a11bb14a" containerID="0e2619c467f0411e941a6a912389786f0929d6d36f5e788504d0e3f99853a2b0" exitCode=0 Nov 03 21:35:55.695985 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.695908 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-62df72ff-64d485b449-25tsr" Nov 03 21:35:55.695985 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.695957 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-62df72ff-64d485b449-25tsr" event={"ID":"ed257ba2-2bf9-4a48-8254-2ec8a11bb14a","Type":"ContainerDied","Data":"0e2619c467f0411e941a6a912389786f0929d6d36f5e788504d0e3f99853a2b0"} Nov 03 21:35:55.696084 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.695997 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-62df72ff-64d485b449-25tsr" event={"ID":"ed257ba2-2bf9-4a48-8254-2ec8a11bb14a","Type":"ContainerDied","Data":"e4698e26a468d042f21d7aafb51dc4253ca974a03e948ec740898e23b22c58f2"} Nov 03 21:35:55.705509 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.705486 4493 scope.go:122] "RemoveContainer" containerID="ea21bbc1505849a6c87cee9a78b57dad5e6abd6f1d41167f9b558e036285f010" Nov 03 21:35:55.705902 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:35:55.705879 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea21bbc1505849a6c87cee9a78b57dad5e6abd6f1d41167f9b558e036285f010\": container with ID starting with ea21bbc1505849a6c87cee9a78b57dad5e6abd6f1d41167f9b558e036285f010 not found: ID does not exist" containerID="ea21bbc1505849a6c87cee9a78b57dad5e6abd6f1d41167f9b558e036285f010" Nov 03 21:35:55.706015 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.705914 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea21bbc1505849a6c87cee9a78b57dad5e6abd6f1d41167f9b558e036285f010"} err="failed to get container status \"ea21bbc1505849a6c87cee9a78b57dad5e6abd6f1d41167f9b558e036285f010\": rpc error: code = NotFound desc = could not find container \"ea21bbc1505849a6c87cee9a78b57dad5e6abd6f1d41167f9b558e036285f010\": container with ID starting with ea21bbc1505849a6c87cee9a78b57dad5e6abd6f1d41167f9b558e036285f010 not found: ID does not exist" Nov 03 21:35:55.706015 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.705939 4493 scope.go:122] "RemoveContainer" containerID="0e2619c467f0411e941a6a912389786f0929d6d36f5e788504d0e3f99853a2b0" Nov 03 21:35:55.713444 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.713420 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed257ba2-2bf9-4a48-8254-2ec8a11bb14a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed257ba2-2bf9-4a48-8254-2ec8a11bb14a-plugins-config-volume\") pod \"ed257ba2-2bf9-4a48-8254-2ec8a11bb14a\" (UID: \"ed257ba2-2bf9-4a48-8254-2ec8a11bb14a\") " Nov 03 21:35:55.713654 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.713637 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed257ba2-2bf9-4a48-8254-2ec8a11bb14a-kube-api-access-94hqd\" (UniqueName: \"kubernetes.io/projected/ed257ba2-2bf9-4a48-8254-2ec8a11bb14a-kube-api-access-94hqd\") pod \"ed257ba2-2bf9-4a48-8254-2ec8a11bb14a\" (UID: \"ed257ba2-2bf9-4a48-8254-2ec8a11bb14a\") " Nov 03 21:35:55.713879 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.713849 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed257ba2-2bf9-4a48-8254-2ec8a11bb14a-plugins-config-volume" pod "ed257ba2-2bf9-4a48-8254-2ec8a11bb14a" (UID: "ed257ba2-2bf9-4a48-8254-2ec8a11bb14a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:35:55.714107 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.714086 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed257ba2-2bf9-4a48-8254-2ec8a11bb14a-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:55.715900 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.715873 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed257ba2-2bf9-4a48-8254-2ec8a11bb14a-kube-api-access-94hqd" pod "ed257ba2-2bf9-4a48-8254-2ec8a11bb14a" (UID: "ed257ba2-2bf9-4a48-8254-2ec8a11bb14a"). InnerVolumeSpecName "kube-api-access-94hqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:35:55.716296 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.716281 4493 scope.go:122] "RemoveContainer" containerID="0e2619c467f0411e941a6a912389786f0929d6d36f5e788504d0e3f99853a2b0" Nov 03 21:35:55.716650 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:35:55.716629 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e2619c467f0411e941a6a912389786f0929d6d36f5e788504d0e3f99853a2b0\": container with ID starting with 0e2619c467f0411e941a6a912389786f0929d6d36f5e788504d0e3f99853a2b0 not found: ID does not exist" containerID="0e2619c467f0411e941a6a912389786f0929d6d36f5e788504d0e3f99853a2b0" Nov 03 21:35:55.716714 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.716659 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2619c467f0411e941a6a912389786f0929d6d36f5e788504d0e3f99853a2b0"} err="failed to get container status \"0e2619c467f0411e941a6a912389786f0929d6d36f5e788504d0e3f99853a2b0\": rpc error: code = NotFound desc = could not find container \"0e2619c467f0411e941a6a912389786f0929d6d36f5e788504d0e3f99853a2b0\": container with ID starting with 0e2619c467f0411e941a6a912389786f0929d6d36f5e788504d0e3f99853a2b0 not found: ID does not exist" Nov 03 21:35:55.814487 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.814451 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6faa2a03-f985-4d4d-a458-58ae4afe04aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6faa2a03-f985-4d4d-a458-58ae4afe04aa-plugins-config-volume\") pod \"6faa2a03-f985-4d4d-a458-58ae4afe04aa\" (UID: \"6faa2a03-f985-4d4d-a458-58ae4afe04aa\") " Nov 03 21:35:55.814696 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.814592 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6faa2a03-f985-4d4d-a458-58ae4afe04aa-kube-api-access-mxxjj\" (UniqueName: \"kubernetes.io/projected/6faa2a03-f985-4d4d-a458-58ae4afe04aa-kube-api-access-mxxjj\") pod \"6faa2a03-f985-4d4d-a458-58ae4afe04aa\" (UID: \"6faa2a03-f985-4d4d-a458-58ae4afe04aa\") " Nov 03 21:35:55.814868 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.814841 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6faa2a03-f985-4d4d-a458-58ae4afe04aa-plugins-config-volume" pod "6faa2a03-f985-4d4d-a458-58ae4afe04aa" (UID: "6faa2a03-f985-4d4d-a458-58ae4afe04aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:35:55.814948 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.814920 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94hqd\" (UniqueName: \"kubernetes.io/projected/ed257ba2-2bf9-4a48-8254-2ec8a11bb14a-kube-api-access-94hqd\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:55.814948 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.814941 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6faa2a03-f985-4d4d-a458-58ae4afe04aa-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:55.816926 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.816906 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6faa2a03-f985-4d4d-a458-58ae4afe04aa-kube-api-access-mxxjj" pod "6faa2a03-f985-4d4d-a458-58ae4afe04aa" (UID: "6faa2a03-f985-4d4d-a458-58ae4afe04aa"). InnerVolumeSpecName "kube-api-access-mxxjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:35:55.915953 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:55.915886 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxxjj\" (UniqueName: \"kubernetes.io/projected/6faa2a03-f985-4d4d-a458-58ae4afe04aa-kube-api-access-mxxjj\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:56.117860 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.117827 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8"] Nov 03 21:35:56.120862 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.120839 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-62df72ff-64f898c99f-bwwz8"] Nov 03 21:35:56.134042 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.134016 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5"] Nov 03 21:35:56.134216 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.134196 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5" podUID="202b5530-4a86-4bbe-a6f7-b41af4da46ca" containerName="maas-api" containerID="cri-o://1dbb49d913192f5bfd21944780ed96c7e57915c04e5132fd13d42e355c9633e6" gracePeriod=30 Nov 03 21:35:56.239608 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.239517 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-62df72ff-64d485b449-25tsr"] Nov 03 21:35:56.242063 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.242028 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-62df72ff-64d485b449-25tsr"] Nov 03 21:35:56.401797 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.401770 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5" Nov 03 21:35:56.521475 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.521404 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/202b5530-4a86-4bbe-a6f7-b41af4da46ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/202b5530-4a86-4bbe-a6f7-b41af4da46ca-metrics-certs\") pod \"202b5530-4a86-4bbe-a6f7-b41af4da46ca\" (UID: \"202b5530-4a86-4bbe-a6f7-b41af4da46ca\") " Nov 03 21:35:56.521475 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.521447 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/202b5530-4a86-4bbe-a6f7-b41af4da46ca-kube-api-access-qnflt\" (UniqueName: \"kubernetes.io/projected/202b5530-4a86-4bbe-a6f7-b41af4da46ca-kube-api-access-qnflt\") pod \"202b5530-4a86-4bbe-a6f7-b41af4da46ca\" (UID: \"202b5530-4a86-4bbe-a6f7-b41af4da46ca\") " Nov 03 21:35:56.521703 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.521485 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/202b5530-4a86-4bbe-a6f7-b41af4da46ca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/202b5530-4a86-4bbe-a6f7-b41af4da46ca-openshift-service-ca\") pod \"202b5530-4a86-4bbe-a6f7-b41af4da46ca\" (UID: \"202b5530-4a86-4bbe-a6f7-b41af4da46ca\") " Nov 03 21:35:56.521703 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.521512 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/202b5530-4a86-4bbe-a6f7-b41af4da46ca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/202b5530-4a86-4bbe-a6f7-b41af4da46ca-maas-api-tls\") pod \"202b5530-4a86-4bbe-a6f7-b41af4da46ca\" (UID: \"202b5530-4a86-4bbe-a6f7-b41af4da46ca\") " Nov 03 21:35:56.521944 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.521913 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202b5530-4a86-4bbe-a6f7-b41af4da46ca-openshift-service-ca" pod "202b5530-4a86-4bbe-a6f7-b41af4da46ca" (UID: "202b5530-4a86-4bbe-a6f7-b41af4da46ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:35:56.524250 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.524229 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/202b5530-4a86-4bbe-a6f7-b41af4da46ca-maas-api-tls" pod "202b5530-4a86-4bbe-a6f7-b41af4da46ca" (UID: "202b5530-4a86-4bbe-a6f7-b41af4da46ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:35:56.524386 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.524248 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/202b5530-4a86-4bbe-a6f7-b41af4da46ca-metrics-certs" pod "202b5530-4a86-4bbe-a6f7-b41af4da46ca" (UID: "202b5530-4a86-4bbe-a6f7-b41af4da46ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:35:56.524386 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.524263 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202b5530-4a86-4bbe-a6f7-b41af4da46ca-kube-api-access-qnflt" pod "202b5530-4a86-4bbe-a6f7-b41af4da46ca" (UID: "202b5530-4a86-4bbe-a6f7-b41af4da46ca"). InnerVolumeSpecName "kube-api-access-qnflt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:35:56.560451 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.560424 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6faa2a03-f985-4d4d-a458-58ae4afe04aa" path="/var/lib/kubelet/pods/6faa2a03-f985-4d4d-a458-58ae4afe04aa/volumes" Nov 03 21:35:56.560883 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.560862 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed257ba2-2bf9-4a48-8254-2ec8a11bb14a" path="/var/lib/kubelet/pods/ed257ba2-2bf9-4a48-8254-2ec8a11bb14a/volumes" Nov 03 21:35:56.622889 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.622855 4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/202b5530-4a86-4bbe-a6f7-b41af4da46ca-openshift-service-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:56.622889 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.622884 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/202b5530-4a86-4bbe-a6f7-b41af4da46ca-maas-api-tls\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:56.622889 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.622894 4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/202b5530-4a86-4bbe-a6f7-b41af4da46ca-metrics-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:56.623118 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.622905 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnflt\" (UniqueName: \"kubernetes.io/projected/202b5530-4a86-4bbe-a6f7-b41af4da46ca-kube-api-access-qnflt\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:56.703414 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.703372 4493 generic.go:356] "Generic (PLEG): container finished" podID="202b5530-4a86-4bbe-a6f7-b41af4da46ca" containerID="1dbb49d913192f5bfd21944780ed96c7e57915c04e5132fd13d42e355c9633e6" exitCode=0 Nov 03 21:35:56.703626 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.703453 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5" event={"ID":"202b5530-4a86-4bbe-a6f7-b41af4da46ca","Type":"ContainerDied","Data":"1dbb49d913192f5bfd21944780ed96c7e57915c04e5132fd13d42e355c9633e6"} Nov 03 21:35:56.703626 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.703480 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5" Nov 03 21:35:56.703626 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.703502 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5" event={"ID":"202b5530-4a86-4bbe-a6f7-b41af4da46ca","Type":"ContainerDied","Data":"e4527888ca9403319b830c97c77757c365ab9e185dabdd7d5052b951c81456ab"} Nov 03 21:35:56.703626 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.703532 4493 scope.go:122] "RemoveContainer" containerID="1dbb49d913192f5bfd21944780ed96c7e57915c04e5132fd13d42e355c9633e6" Nov 03 21:35:56.712176 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.712158 4493 scope.go:122] "RemoveContainer" containerID="1dbb49d913192f5bfd21944780ed96c7e57915c04e5132fd13d42e355c9633e6" Nov 03 21:35:56.712468 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:35:56.712447 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dbb49d913192f5bfd21944780ed96c7e57915c04e5132fd13d42e355c9633e6\": container with ID starting with 1dbb49d913192f5bfd21944780ed96c7e57915c04e5132fd13d42e355c9633e6 not found: ID does not exist" containerID="1dbb49d913192f5bfd21944780ed96c7e57915c04e5132fd13d42e355c9633e6" Nov 03 21:35:56.712551 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.712476 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dbb49d913192f5bfd21944780ed96c7e57915c04e5132fd13d42e355c9633e6"} err="failed to get container status \"1dbb49d913192f5bfd21944780ed96c7e57915c04e5132fd13d42e355c9633e6\": rpc error: code = NotFound desc = could not find container \"1dbb49d913192f5bfd21944780ed96c7e57915c04e5132fd13d42e355c9633e6\": container with ID starting with 1dbb49d913192f5bfd21944780ed96c7e57915c04e5132fd13d42e355c9633e6 not found: ID does not exist" Nov 03 21:35:56.871055 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.871024 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5"] Nov 03 21:35:56.875367 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:56.875321 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62df72ff-8445f4c7f4-chrm5"] Nov 03 21:35:58.560603 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:58.560566 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="202b5530-4a86-4bbe-a6f7-b41af4da46ca" path="/var/lib/kubelet/pods/202b5530-4a86-4bbe-a6f7-b41af4da46ca/volumes" Nov 03 21:35:59.873482 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:59.873443 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj"] Nov 03 21:35:59.873879 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:35:59.873666 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" podUID="7f4990b9-156a-4777-a11c-7b470d725903" containerName="istio-proxy" containerID="cri-o://490e1e545d7f9c806ae3f4d02c78d13f33c54c8a0aecde7424ba61068f6a3bb7" gracePeriod=30 Nov 03 21:36:01.510764 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.510732 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2"] Nov 03 21:36:01.511173 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.511113 4493 cpu_manager.go:414] "containerMap: removing container" podUID="6faa2a03-f985-4d4d-a458-58ae4afe04aa" containerName="payload-pre-processing" Nov 03 21:36:01.511173 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.511124 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6faa2a03-f985-4d4d-a458-58ae4afe04aa" containerName="payload-pre-processing" Nov 03 21:36:01.511173 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.511136 4493 cpu_manager.go:414] "containerMap: removing container" podUID="202b5530-4a86-4bbe-a6f7-b41af4da46ca" containerName="maas-api" Nov 03 21:36:01.511173 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.511143 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="202b5530-4a86-4bbe-a6f7-b41af4da46ca" containerName="maas-api" Nov 03 21:36:01.511173 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.511151 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ed257ba2-2bf9-4a48-8254-2ec8a11bb14a" containerName="payload-processing" Nov 03 21:36:01.511173 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.511155 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed257ba2-2bf9-4a48-8254-2ec8a11bb14a" containerName="payload-processing" Nov 03 21:36:01.511392 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.511238 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6faa2a03-f985-4d4d-a458-58ae4afe04aa" containerName="payload-pre-processing" Nov 03 21:36:01.511392 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.511246 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="202b5530-4a86-4bbe-a6f7-b41af4da46ca" containerName="maas-api" Nov 03 21:36:01.511392 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.511254 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed257ba2-2bf9-4a48-8254-2ec8a11bb14a" containerName="payload-processing" Nov 03 21:36:01.515827 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.515809 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.569373 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.569311 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-envoy\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.569558 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.569440 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9vcv\" (UniqueName: \"kubernetes.io/projected/c35a8aed-f40e-48dd-9138-19f5ce831a33-kube-api-access-j9vcv\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.569558 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.569470 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-workload-certs\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.569637 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.569604 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c35a8aed-f40e-48dd-9138-19f5ce831a33-istiod-ca-cert\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.569741 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.569724 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-credential-socket\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.569786 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.569772 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-token\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.569825 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.569801 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-workload-socket\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.569894 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.569879 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-data\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.569953 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.569941 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-podinfo\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.611624 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.611587 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2"] Nov 03 21:36:01.625530 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.625502 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-1850fc-openshift-default-dockercfg-qj8fc\"" Nov 03 21:36:01.671180 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.671081 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9vcv\" (UniqueName: \"kubernetes.io/projected/c35a8aed-f40e-48dd-9138-19f5ce831a33-kube-api-access-j9vcv\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.671180 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.671162 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-workload-certs\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.671744 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.671718 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-workload-certs\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.672471 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.672418 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c35a8aed-f40e-48dd-9138-19f5ce831a33-istiod-ca-cert\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.672627 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.672539 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c35a8aed-f40e-48dd-9138-19f5ce831a33-istiod-ca-cert\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.672627 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.672540 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-credential-socket\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.672769 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.672634 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-token\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.672769 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.672660 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-workload-socket\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.672769 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.672687 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-data\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.672969 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.672810 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-credential-socket\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.672969 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.672897 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-podinfo\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.672969 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.672926 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-envoy\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.673143 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.673066 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-data\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.673603 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.673579 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-workload-socket\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.676197 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.676170 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-podinfo\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.676414 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.676394 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-envoy\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.731977 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.731943 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-token\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.732162 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.732055 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9vcv\" (UniqueName: \"kubernetes.io/projected/c35a8aed-f40e-48dd-9138-19f5ce831a33-kube-api-access-j9vcv\") pod \"e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:01.828497 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:01.828466 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:02.261090 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:02.261065 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2"] Nov 03 21:36:02.261788 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:36:02.261764 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35a8aed_f40e_48dd_9138_19f5ce831a33.slice/crio-6b6f76962e56987708516dcdfcd072efbacddfc8d6020f111fb5eeb01cbb2203 WatchSource:0}: Error finding container 6b6f76962e56987708516dcdfcd072efbacddfc8d6020f111fb5eeb01cbb2203: Status 404 returned error can't find the container with id 6b6f76962e56987708516dcdfcd072efbacddfc8d6020f111fb5eeb01cbb2203 Nov 03 21:36:02.264216 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:02.264179 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:36:02.264378 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:02.264248 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:36:02.264378 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:02.264284 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:36:02.737821 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:02.737784 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" event={"ID":"c35a8aed-f40e-48dd-9138-19f5ce831a33","Type":"ContainerStarted","Data":"556376d767803c302785632b97a2141dfb0e3727e6ebd9968909985a771295f1"} Nov 03 21:36:02.738297 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:02.737881 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" event={"ID":"c35a8aed-f40e-48dd-9138-19f5ce831a33","Type":"ContainerStarted","Data":"6b6f76962e56987708516dcdfcd072efbacddfc8d6020f111fb5eeb01cbb2203"} Nov 03 21:36:02.829555 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:02.829519 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:02.857719 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:02.857675 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" podStartSLOduration=1.857661937 podStartE2EDuration="1.857661937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:36:02.855747881 +0000 UTC m=+4034.413738381" watchObservedRunningTime="2026-11-03 21:36:02.857661937 +0000 UTC m=+4034.415652435" Nov 03 21:36:03.834402 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:03.834376 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:04.747658 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:04.747631 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:04.748745 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:04.748726 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:05.131416 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.131395 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:36:05.206829 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.206802 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f4990b9-156a-4777-a11c-7b470d725903-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4990b9-156a-4777-a11c-7b470d725903-istio-podinfo\") pod \"7f4990b9-156a-4777-a11c-7b470d725903\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " Nov 03 21:36:05.206941 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.206854 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-istio-envoy\") pod \"7f4990b9-156a-4777-a11c-7b470d725903\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " Nov 03 21:36:05.206941 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.206884 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f4990b9-156a-4777-a11c-7b470d725903-kube-api-access-pwhtf\" (UniqueName: \"kubernetes.io/projected/7f4990b9-156a-4777-a11c-7b470d725903-kube-api-access-pwhtf\") pod \"7f4990b9-156a-4777-a11c-7b470d725903\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " Nov 03 21:36:05.206941 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.206905 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f4990b9-156a-4777-a11c-7b470d725903-istio-token\" (UniqueName: \"kubernetes.io/projected/7f4990b9-156a-4777-a11c-7b470d725903-istio-token\") pod \"7f4990b9-156a-4777-a11c-7b470d725903\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " Nov 03 21:36:05.207100 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.206933 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-credential-socket\") pod \"7f4990b9-156a-4777-a11c-7b470d725903\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " Nov 03 21:36:05.207100 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.206968 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-istio-data\") pod \"7f4990b9-156a-4777-a11c-7b470d725903\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " Nov 03 21:36:05.207970 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.207283 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-credential-socket" pod "7f4990b9-156a-4777-a11c-7b470d725903" (UID: "7f4990b9-156a-4777-a11c-7b470d725903"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:36:05.207970 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.207298 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-istio-data" pod "7f4990b9-156a-4777-a11c-7b470d725903" (UID: "7f4990b9-156a-4777-a11c-7b470d725903"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:36:05.207970 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.207285 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f4990b9-156a-4777-a11c-7b470d725903-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4990b9-156a-4777-a11c-7b470d725903-istiod-ca-cert\") pod \"7f4990b9-156a-4777-a11c-7b470d725903\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " Nov 03 21:36:05.207970 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.207391 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-workload-socket\") pod \"7f4990b9-156a-4777-a11c-7b470d725903\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " Nov 03 21:36:05.207970 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.207505 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-workload-certs\") pod \"7f4990b9-156a-4777-a11c-7b470d725903\" (UID: \"7f4990b9-156a-4777-a11c-7b470d725903\") " Nov 03 21:36:05.207970 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.207568 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f4990b9-156a-4777-a11c-7b470d725903-istiod-ca-cert" pod "7f4990b9-156a-4777-a11c-7b470d725903" (UID: "7f4990b9-156a-4777-a11c-7b470d725903"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:36:05.207970 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.207906 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:05.207970 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.207918 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-workload-certs" pod "7f4990b9-156a-4777-a11c-7b470d725903" (UID: "7f4990b9-156a-4777-a11c-7b470d725903"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:36:05.207970 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.207927 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:05.207970 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.207945 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-workload-socket" pod "7f4990b9-156a-4777-a11c-7b470d725903" (UID: "7f4990b9-156a-4777-a11c-7b470d725903"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:36:05.208399 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.207958 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4990b9-156a-4777-a11c-7b470d725903-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:05.209218 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.209189 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-istio-envoy" pod "7f4990b9-156a-4777-a11c-7b470d725903" (UID: "7f4990b9-156a-4777-a11c-7b470d725903"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:36:05.209509 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.209487 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f4990b9-156a-4777-a11c-7b470d725903-istio-podinfo" pod "7f4990b9-156a-4777-a11c-7b470d725903" (UID: "7f4990b9-156a-4777-a11c-7b470d725903"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:36:05.209776 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.209757 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f4990b9-156a-4777-a11c-7b470d725903-kube-api-access-pwhtf" pod "7f4990b9-156a-4777-a11c-7b470d725903" (UID: "7f4990b9-156a-4777-a11c-7b470d725903"). InnerVolumeSpecName "kube-api-access-pwhtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:36:05.209844 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.209780 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f4990b9-156a-4777-a11c-7b470d725903-istio-token" pod "7f4990b9-156a-4777-a11c-7b470d725903" (UID: "7f4990b9-156a-4777-a11c-7b470d725903"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:36:05.309414 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.309383 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:05.309414 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.309414 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4990b9-156a-4777-a11c-7b470d725903-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:05.309616 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.309424 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:05.309616 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.309433 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwhtf\" (UniqueName: \"kubernetes.io/projected/7f4990b9-156a-4777-a11c-7b470d725903-kube-api-access-pwhtf\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:05.309616 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.309443 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f4990b9-156a-4777-a11c-7b470d725903-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:05.309616 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.309452 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4990b9-156a-4777-a11c-7b470d725903-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:05.754747 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.754663 4493 generic.go:356] "Generic (PLEG): container finished" podID="7f4990b9-156a-4777-a11c-7b470d725903" containerID="490e1e545d7f9c806ae3f4d02c78d13f33c54c8a0aecde7424ba61068f6a3bb7" exitCode=0 Nov 03 21:36:05.754747 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.754731 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" Nov 03 21:36:05.754967 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.754744 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" event={"ID":"7f4990b9-156a-4777-a11c-7b470d725903","Type":"ContainerDied","Data":"490e1e545d7f9c806ae3f4d02c78d13f33c54c8a0aecde7424ba61068f6a3bb7"} Nov 03 21:36:05.754967 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.754793 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj" event={"ID":"7f4990b9-156a-4777-a11c-7b470d725903","Type":"ContainerDied","Data":"70a3d540adba0089b9e4577135e8868bb0df6de6b7c37093d94e612df599478a"} Nov 03 21:36:05.754967 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.754815 4493 scope.go:122] "RemoveContainer" containerID="490e1e545d7f9c806ae3f4d02c78d13f33c54c8a0aecde7424ba61068f6a3bb7" Nov 03 21:36:05.764620 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.764602 4493 scope.go:122] "RemoveContainer" containerID="5f5116c05e288ff743426ac79cadb1d2f11f6e6fec6cc008f7d0147fa1d042fa" Nov 03 21:36:05.772662 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.772644 4493 scope.go:122] "RemoveContainer" containerID="490e1e545d7f9c806ae3f4d02c78d13f33c54c8a0aecde7424ba61068f6a3bb7" Nov 03 21:36:05.772923 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:36:05.772896 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"490e1e545d7f9c806ae3f4d02c78d13f33c54c8a0aecde7424ba61068f6a3bb7\": container with ID starting with 490e1e545d7f9c806ae3f4d02c78d13f33c54c8a0aecde7424ba61068f6a3bb7 not found: ID does not exist" containerID="490e1e545d7f9c806ae3f4d02c78d13f33c54c8a0aecde7424ba61068f6a3bb7" Nov 03 21:36:05.772971 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.772935 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"490e1e545d7f9c806ae3f4d02c78d13f33c54c8a0aecde7424ba61068f6a3bb7"} err="failed to get container status \"490e1e545d7f9c806ae3f4d02c78d13f33c54c8a0aecde7424ba61068f6a3bb7\": rpc error: code = NotFound desc = could not find container \"490e1e545d7f9c806ae3f4d02c78d13f33c54c8a0aecde7424ba61068f6a3bb7\": container with ID starting with 490e1e545d7f9c806ae3f4d02c78d13f33c54c8a0aecde7424ba61068f6a3bb7 not found: ID does not exist" Nov 03 21:36:05.772971 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.772957 4493 scope.go:122] "RemoveContainer" containerID="5f5116c05e288ff743426ac79cadb1d2f11f6e6fec6cc008f7d0147fa1d042fa" Nov 03 21:36:05.773214 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:36:05.773194 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f5116c05e288ff743426ac79cadb1d2f11f6e6fec6cc008f7d0147fa1d042fa\": container with ID starting with 5f5116c05e288ff743426ac79cadb1d2f11f6e6fec6cc008f7d0147fa1d042fa not found: ID does not exist" containerID="5f5116c05e288ff743426ac79cadb1d2f11f6e6fec6cc008f7d0147fa1d042fa" Nov 03 21:36:05.773260 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.773224 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f5116c05e288ff743426ac79cadb1d2f11f6e6fec6cc008f7d0147fa1d042fa"} err="failed to get container status \"5f5116c05e288ff743426ac79cadb1d2f11f6e6fec6cc008f7d0147fa1d042fa\": rpc error: code = NotFound desc = could not find container \"5f5116c05e288ff743426ac79cadb1d2f11f6e6fec6cc008f7d0147fa1d042fa\": container with ID starting with 5f5116c05e288ff743426ac79cadb1d2f11f6e6fec6cc008f7d0147fa1d042fa not found: ID does not exist" Nov 03 21:36:05.924310 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.924279 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj"] Nov 03 21:36:05.976152 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:05.976124 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-62df72ff-openshift-default-dcb74987-574xj"] Nov 03 21:36:06.561255 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:06.561219 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f4990b9-156a-4777-a11c-7b470d725903" path="/var/lib/kubelet/pods/7f4990b9-156a-4777-a11c-7b470d725903/volumes" Nov 03 21:36:09.222689 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.222657 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc"] Nov 03 21:36:09.223238 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.223218 4493 cpu_manager.go:414] "containerMap: removing container" podUID="7f4990b9-156a-4777-a11c-7b470d725903" containerName="istio-proxy" Nov 03 21:36:09.223326 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.223241 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f4990b9-156a-4777-a11c-7b470d725903" containerName="istio-proxy" Nov 03 21:36:09.223423 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.223407 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f4990b9-156a-4777-a11c-7b470d725903" containerName="istio-proxy" Nov 03 21:36:09.223486 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.223427 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f4990b9-156a-4777-a11c-7b470d725903" containerName="istio-proxy" Nov 03 21:36:09.228050 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.228027 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc" Nov 03 21:36:09.245215 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.245190 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sth6m\" (UniqueName: \"kubernetes.io/projected/ec32775d-1e45-4d07-8422-f47cd77e116d-kube-api-access-sth6m\") pod \"payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc\" (UID: \"ec32775d-1e45-4d07-8422-f47cd77e116d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc" Nov 03 21:36:09.245372 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.245326 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec32775d-1e45-4d07-8422-f47cd77e116d-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc\" (UID: \"ec32775d-1e45-4d07-8422-f47cd77e116d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc" Nov 03 21:36:09.278948 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.278923 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc"] Nov 03 21:36:09.328792 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.328758 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-1850fc\"" Nov 03 21:36:09.328792 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.328777 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-1850fc-dockercfg-4tdhl\"" Nov 03 21:36:09.346904 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.346861 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sth6m\" (UniqueName: \"kubernetes.io/projected/ec32775d-1e45-4d07-8422-f47cd77e116d-kube-api-access-sth6m\") pod \"payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc\" (UID: \"ec32775d-1e45-4d07-8422-f47cd77e116d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc" Nov 03 21:36:09.347113 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.346979 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec32775d-1e45-4d07-8422-f47cd77e116d-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc\" (UID: \"ec32775d-1e45-4d07-8422-f47cd77e116d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc" Nov 03 21:36:09.347945 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.347912 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec32775d-1e45-4d07-8422-f47cd77e116d-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc\" (UID: \"ec32775d-1e45-4d07-8422-f47cd77e116d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc" Nov 03 21:36:09.408600 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.408568 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq"] Nov 03 21:36:09.409044 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.409025 4493 cpu_manager.go:414] "containerMap: removing container" podUID="7f4990b9-156a-4777-a11c-7b470d725903" containerName="istio-proxy" Nov 03 21:36:09.409139 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.409047 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f4990b9-156a-4777-a11c-7b470d725903" containerName="istio-proxy" Nov 03 21:36:09.410305 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.410279 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sth6m\" (UniqueName: \"kubernetes.io/projected/ec32775d-1e45-4d07-8422-f47cd77e116d-kube-api-access-sth6m\") pod \"payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc\" (UID: \"ec32775d-1e45-4d07-8422-f47cd77e116d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc" Nov 03 21:36:09.412652 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.412625 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq" Nov 03 21:36:09.447790 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.447766 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4hmz\" (UniqueName: \"kubernetes.io/projected/01cddd9d-7cdf-4e9f-8828-525272f3b558-kube-api-access-f4hmz\") pod \"payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq\" (UID: \"01cddd9d-7cdf-4e9f-8828-525272f3b558\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq" Nov 03 21:36:09.447901 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.447802 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01cddd9d-7cdf-4e9f-8828-525272f3b558-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq\" (UID: \"01cddd9d-7cdf-4e9f-8828-525272f3b558\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq" Nov 03 21:36:09.456858 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.456833 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq"] Nov 03 21:36:09.540948 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.540868 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc" Nov 03 21:36:09.549088 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.549063 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4hmz\" (UniqueName: \"kubernetes.io/projected/01cddd9d-7cdf-4e9f-8828-525272f3b558-kube-api-access-f4hmz\") pod \"payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq\" (UID: \"01cddd9d-7cdf-4e9f-8828-525272f3b558\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq" Nov 03 21:36:09.549217 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.549109 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01cddd9d-7cdf-4e9f-8828-525272f3b558-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq\" (UID: \"01cddd9d-7cdf-4e9f-8828-525272f3b558\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq" Nov 03 21:36:09.549742 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.549722 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01cddd9d-7cdf-4e9f-8828-525272f3b558-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq\" (UID: \"01cddd9d-7cdf-4e9f-8828-525272f3b558\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq" Nov 03 21:36:09.609759 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.609733 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4hmz\" (UniqueName: \"kubernetes.io/projected/01cddd9d-7cdf-4e9f-8828-525272f3b558-kube-api-access-f4hmz\") pod \"payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq\" (UID: \"01cddd9d-7cdf-4e9f-8828-525272f3b558\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq" Nov 03 21:36:09.730291 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.730261 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq" Nov 03 21:36:09.913484 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:09.913459 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc"] Nov 03 21:36:09.914573 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:36:09.914531 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec32775d_1e45_4d07_8422_f47cd77e116d.slice/crio-f433939d8954878b9fa441023c857823de824e12ef44b8c3756017b09819dc90 WatchSource:0}: Error finding container f433939d8954878b9fa441023c857823de824e12ef44b8c3756017b09819dc90: Status 404 returned error can't find the container with id f433939d8954878b9fa441023c857823de824e12ef44b8c3756017b09819dc90 Nov 03 21:36:10.103572 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:10.103546 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq"] Nov 03 21:36:10.104306 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:36:10.104286 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01cddd9d_7cdf_4e9f_8828_525272f3b558.slice/crio-f316e028b263640cddfb1a691111f3e3819cd81547e7ba1e757ffac1ed54939e WatchSource:0}: Error finding container f316e028b263640cddfb1a691111f3e3819cd81547e7ba1e757ffac1ed54939e: Status 404 returned error can't find the container with id f316e028b263640cddfb1a691111f3e3819cd81547e7ba1e757ffac1ed54939e Nov 03 21:36:10.784521 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:10.784488 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc" event={"ID":"ec32775d-1e45-4d07-8422-f47cd77e116d","Type":"ContainerStarted","Data":"c50f483e0ace592ceff8af3e9c80add5677055efb6b4550aace8c7fc89271bd8"} Nov 03 21:36:10.785019 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:10.784539 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc" event={"ID":"ec32775d-1e45-4d07-8422-f47cd77e116d","Type":"ContainerStarted","Data":"f433939d8954878b9fa441023c857823de824e12ef44b8c3756017b09819dc90"} Nov 03 21:36:10.785019 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:10.784588 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc" Nov 03 21:36:10.786057 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:10.786033 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq" event={"ID":"01cddd9d-7cdf-4e9f-8828-525272f3b558","Type":"ContainerStarted","Data":"41c884f0370d9860c2235f4ee60f1e2238f7d08874b4cae716a3ce1f84a4a0d2"} Nov 03 21:36:10.786179 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:10.786073 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq" Nov 03 21:36:10.786179 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:10.786089 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq" event={"ID":"01cddd9d-7cdf-4e9f-8828-525272f3b558","Type":"ContainerStarted","Data":"f316e028b263640cddfb1a691111f3e3819cd81547e7ba1e757ffac1ed54939e"} Nov 03 21:36:10.921914 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:10.921869 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc" podStartSLOduration=1.921856049 podStartE2EDuration="1.921856049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:36:10.90031966 +0000 UTC m=+4042.458310159" watchObservedRunningTime="2026-11-03 21:36:10.921856049 +0000 UTC m=+4042.479846547" Nov 03 21:36:11.018616 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:11.018562 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq" podStartSLOduration=2.018545309 podStartE2EDuration="2.018545309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:36:11.016469416 +0000 UTC m=+4042.574459935" watchObservedRunningTime="2026-11-03 21:36:11.018545309 +0000 UTC m=+4042.576535808" Nov 03 21:36:11.498324 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:11.498284 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h"] Nov 03 21:36:11.501971 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:11.501954 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h" Nov 03 21:36:11.556159 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:11.556133 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-1850fc-serving-cert\"" Nov 03 21:36:11.556710 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:11.556686 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h"] Nov 03 21:36:11.569705 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:11.569677 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e40f1881-b555-4ea8-84a9-cb323a0a106a-metrics-certs\") pod \"maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h\" (UID: \"e40f1881-b555-4ea8-84a9-cb323a0a106a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h" Nov 03 21:36:11.569836 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:11.569712 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5dj4\" (UniqueName: \"kubernetes.io/projected/e40f1881-b555-4ea8-84a9-cb323a0a106a-kube-api-access-z5dj4\") pod \"maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h\" (UID: \"e40f1881-b555-4ea8-84a9-cb323a0a106a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h" Nov 03 21:36:11.569894 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:11.569850 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e40f1881-b555-4ea8-84a9-cb323a0a106a-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h\" (UID: \"e40f1881-b555-4ea8-84a9-cb323a0a106a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h" Nov 03 21:36:11.569949 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:11.569921 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e40f1881-b555-4ea8-84a9-cb323a0a106a-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h\" (UID: \"e40f1881-b555-4ea8-84a9-cb323a0a106a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h" Nov 03 21:36:11.671439 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:11.671389 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e40f1881-b555-4ea8-84a9-cb323a0a106a-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h\" (UID: \"e40f1881-b555-4ea8-84a9-cb323a0a106a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h" Nov 03 21:36:11.671731 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:11.671562 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e40f1881-b555-4ea8-84a9-cb323a0a106a-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h\" (UID: \"e40f1881-b555-4ea8-84a9-cb323a0a106a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h" Nov 03 21:36:11.671731 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:11.671673 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e40f1881-b555-4ea8-84a9-cb323a0a106a-metrics-certs\") pod \"maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h\" (UID: \"e40f1881-b555-4ea8-84a9-cb323a0a106a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h" Nov 03 21:36:11.671731 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:11.671723 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5dj4\" (UniqueName: \"kubernetes.io/projected/e40f1881-b555-4ea8-84a9-cb323a0a106a-kube-api-access-z5dj4\") pod \"maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h\" (UID: \"e40f1881-b555-4ea8-84a9-cb323a0a106a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h" Nov 03 21:36:11.671894 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:36:11.671730 4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-serving-cert: secret "maas-api-e2e-authcfg-http-1850fc-serving-cert" not found Nov 03 21:36:11.671894 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:36:11.671799 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e40f1881-b555-4ea8-84a9-cb323a0a106a-maas-api-tls podName:e40f1881-b555-4ea8-84a9-cb323a0a106a nodeName:}" failed. No retries permitted until 2026-11-03 21:36:12.17177835 +0000 UTC m=+4043.729768842 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e40f1881-b555-4ea8-84a9-cb323a0a106a-maas-api-tls") pod "maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h" (UID: "e40f1881-b555-4ea8-84a9-cb323a0a106a") : secret "maas-api-e2e-authcfg-http-1850fc-serving-cert" not found Nov 03 21:36:11.672195 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:11.672174 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e40f1881-b555-4ea8-84a9-cb323a0a106a-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h\" (UID: \"e40f1881-b555-4ea8-84a9-cb323a0a106a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h" Nov 03 21:36:11.674276 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:11.674257 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e40f1881-b555-4ea8-84a9-cb323a0a106a-metrics-certs\") pod \"maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h\" (UID: \"e40f1881-b555-4ea8-84a9-cb323a0a106a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h" Nov 03 21:36:11.731786 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:11.731759 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5dj4\" (UniqueName: \"kubernetes.io/projected/e40f1881-b555-4ea8-84a9-cb323a0a106a-kube-api-access-z5dj4\") pod \"maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h\" (UID: \"e40f1881-b555-4ea8-84a9-cb323a0a106a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h" Nov 03 21:36:12.179423 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:12.179393 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e40f1881-b555-4ea8-84a9-cb323a0a106a-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h\" (UID: \"e40f1881-b555-4ea8-84a9-cb323a0a106a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h" Nov 03 21:36:12.182040 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:12.182015 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e40f1881-b555-4ea8-84a9-cb323a0a106a-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h\" (UID: \"e40f1881-b555-4ea8-84a9-cb323a0a106a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h" Nov 03 21:36:12.415212 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:12.415177 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h" Nov 03 21:36:12.788814 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:12.788789 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h"] Nov 03 21:36:12.789625 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:36:12.789586 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode40f1881_b555_4ea8_84a9_cb323a0a106a.slice/crio-d6ccfafc809b840c3ff57ffdb88d7b8aaaefffc6c8031597fb2d3aefde395fd5 WatchSource:0}: Error finding container d6ccfafc809b840c3ff57ffdb88d7b8aaaefffc6c8031597fb2d3aefde395fd5: Status 404 returned error can't find the container with id d6ccfafc809b840c3ff57ffdb88d7b8aaaefffc6c8031597fb2d3aefde395fd5 Nov 03 21:36:12.795782 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:12.795758 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h" event={"ID":"e40f1881-b555-4ea8-84a9-cb323a0a106a","Type":"ContainerStarted","Data":"d6ccfafc809b840c3ff57ffdb88d7b8aaaefffc6c8031597fb2d3aefde395fd5"} Nov 03 21:36:13.802616 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:13.802576 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h" event={"ID":"e40f1881-b555-4ea8-84a9-cb323a0a106a","Type":"ContainerStarted","Data":"24b5ad914dfe4305119c885f9eb7f896ed61adbe877295039e1593aba530b343"} Nov 03 21:36:13.803060 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:13.802704 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h" Nov 03 21:36:13.934007 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:13.933947 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h" podStartSLOduration=2.018848438 podStartE2EDuration="2.933934219s" totalImagesPullingTime="915.085781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:36:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:36:13.933066807 +0000 UTC m=+4045.491057306" watchObservedRunningTime="2026-11-03 21:36:13.933934219 +0000 UTC m=+4045.491924718" Nov 03 21:36:19.814130 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:19.814102 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h" Nov 03 21:36:21.744073 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:36:21.744032 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:36:21.808879 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:21.808843 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq" Nov 03 21:36:21.809055 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:21.808928 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc" Nov 03 21:36:28.880295 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:28.880260 4493 generic.go:356] "Generic (PLEG): container finished" podID="c35a8aed-f40e-48dd-9138-19f5ce831a33" containerID="556376d767803c302785632b97a2141dfb0e3727e6ebd9968909985a771295f1" exitCode=0 Nov 03 21:36:28.880746 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:28.880327 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" event={"ID":"c35a8aed-f40e-48dd-9138-19f5ce831a33","Type":"ContainerDied","Data":"556376d767803c302785632b97a2141dfb0e3727e6ebd9968909985a771295f1"} Nov 03 21:36:28.880746 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:28.880681 4493 scope.go:122] "RemoveContainer" containerID="556376d767803c302785632b97a2141dfb0e3727e6ebd9968909985a771295f1" Nov 03 21:36:28.881976 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:28.881950 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:36:28.882038 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:28.882020 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:36:28.882075 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:28.882055 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:36:29.910705 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:29.910670 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" event={"ID":"c35a8aed-f40e-48dd-9138-19f5ce831a33","Type":"ContainerStarted","Data":"f8175a7038b4c22ebcbc6b9b81656701791ca87f68bbbb6e998f0e879ab18901"} Nov 03 21:36:30.828755 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:30.828716 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:30.830274 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:30.830244 4493 patch_prober.go:29] interesting pod/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 03 21:36:30.830437 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:30.830294 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" podUID="c35a8aed-f40e-48dd-9138-19f5ce831a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 03 21:36:31.830284 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:31.830248 4493 patch_prober.go:29] interesting pod/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 03 21:36:31.830751 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:31.830309 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" podUID="c35a8aed-f40e-48dd-9138-19f5ce831a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 03 21:36:32.828898 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:32.828860 4493 patch_prober.go:29] interesting pod/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 03 21:36:32.829101 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:32.828921 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" podUID="c35a8aed-f40e-48dd-9138-19f5ce831a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 03 21:36:33.829716 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:33.829682 4493 patch_prober.go:29] interesting pod/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 03 21:36:33.830131 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:33.829742 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" podUID="c35a8aed-f40e-48dd-9138-19f5ce831a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 03 21:36:34.748171 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:34.748130 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:34.829418 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:34.829384 4493 patch_prober.go:29] interesting pod/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 03 21:36:34.829596 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:34.829443 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" podUID="c35a8aed-f40e-48dd-9138-19f5ce831a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 03 21:36:35.833113 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:35.833078 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:35.834189 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:36:35.834173 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:36:51.745037 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:36:51.744998 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:37:21.743427 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:37:21.743393 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:37:51.743787 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:37:51.743750 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:38:04.706493 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:04.706446 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc"] Nov 03 21:38:04.814614 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:04.814584 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq"] Nov 03 21:38:09.706793 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:09.706733 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc" podUID="ec32775d-1e45-4d07-8422-f47cd77e116d" containerName="payload-processing" containerID="cri-o://c50f483e0ace592ceff8af3e9c80add5677055efb6b4550aace8c7fc89271bd8" gracePeriod=25 Nov 03 21:38:09.815680 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:09.815616 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq" podUID="01cddd9d-7cdf-4e9f-8828-525272f3b558" containerName="payload-pre-processing" containerID="cri-o://41c884f0370d9860c2235f4ee60f1e2238f7d08874b4cae716a3ce1f84a4a0d2" gracePeriod=25 Nov 03 21:38:09.959618 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:09.959539 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc" Nov 03 21:38:10.058137 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.058113 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq" Nov 03 21:38:10.096435 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.096409 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec32775d-1e45-4d07-8422-f47cd77e116d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec32775d-1e45-4d07-8422-f47cd77e116d-plugins-config-volume\") pod \"ec32775d-1e45-4d07-8422-f47cd77e116d\" (UID: \"ec32775d-1e45-4d07-8422-f47cd77e116d\") " Nov 03 21:38:10.096594 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.096518 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec32775d-1e45-4d07-8422-f47cd77e116d-kube-api-access-sth6m\" (UniqueName: \"kubernetes.io/projected/ec32775d-1e45-4d07-8422-f47cd77e116d-kube-api-access-sth6m\") pod \"ec32775d-1e45-4d07-8422-f47cd77e116d\" (UID: \"ec32775d-1e45-4d07-8422-f47cd77e116d\") " Nov 03 21:38:10.096779 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.096758 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec32775d-1e45-4d07-8422-f47cd77e116d-plugins-config-volume" pod "ec32775d-1e45-4d07-8422-f47cd77e116d" (UID: "ec32775d-1e45-4d07-8422-f47cd77e116d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:38:10.098760 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.098740 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec32775d-1e45-4d07-8422-f47cd77e116d-kube-api-access-sth6m" pod "ec32775d-1e45-4d07-8422-f47cd77e116d" (UID: "ec32775d-1e45-4d07-8422-f47cd77e116d"). InnerVolumeSpecName "kube-api-access-sth6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:38:10.197643 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.197617 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01cddd9d-7cdf-4e9f-8828-525272f3b558-kube-api-access-f4hmz\" (UniqueName: \"kubernetes.io/projected/01cddd9d-7cdf-4e9f-8828-525272f3b558-kube-api-access-f4hmz\") pod \"01cddd9d-7cdf-4e9f-8828-525272f3b558\" (UID: \"01cddd9d-7cdf-4e9f-8828-525272f3b558\") " Nov 03 21:38:10.197803 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.197724 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01cddd9d-7cdf-4e9f-8828-525272f3b558-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01cddd9d-7cdf-4e9f-8828-525272f3b558-plugins-config-volume\") pod \"01cddd9d-7cdf-4e9f-8828-525272f3b558\" (UID: \"01cddd9d-7cdf-4e9f-8828-525272f3b558\") " Nov 03 21:38:10.198016 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.197994 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sth6m\" (UniqueName: \"kubernetes.io/projected/ec32775d-1e45-4d07-8422-f47cd77e116d-kube-api-access-sth6m\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:10.198061 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.198026 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec32775d-1e45-4d07-8422-f47cd77e116d-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:10.198120 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.198105 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01cddd9d-7cdf-4e9f-8828-525272f3b558-plugins-config-volume" pod "01cddd9d-7cdf-4e9f-8828-525272f3b558" (UID: "01cddd9d-7cdf-4e9f-8828-525272f3b558"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:38:10.200118 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.200083 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01cddd9d-7cdf-4e9f-8828-525272f3b558-kube-api-access-f4hmz" pod "01cddd9d-7cdf-4e9f-8828-525272f3b558" (UID: "01cddd9d-7cdf-4e9f-8828-525272f3b558"). InnerVolumeSpecName "kube-api-access-f4hmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:38:10.299169 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.299141 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01cddd9d-7cdf-4e9f-8828-525272f3b558-plugins-config-volume\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:10.299169 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.299166 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4hmz\" (UniqueName: \"kubernetes.io/projected/01cddd9d-7cdf-4e9f-8828-525272f3b558-kube-api-access-f4hmz\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:10.423038 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.423003 4493 generic.go:356] "Generic (PLEG): container finished" podID="ec32775d-1e45-4d07-8422-f47cd77e116d" containerID="c50f483e0ace592ceff8af3e9c80add5677055efb6b4550aace8c7fc89271bd8" exitCode=0 Nov 03 21:38:10.423205 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.423044 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc" Nov 03 21:38:10.423205 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.423087 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc" event={"ID":"ec32775d-1e45-4d07-8422-f47cd77e116d","Type":"ContainerDied","Data":"c50f483e0ace592ceff8af3e9c80add5677055efb6b4550aace8c7fc89271bd8"} Nov 03 21:38:10.423205 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.423171 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc" event={"ID":"ec32775d-1e45-4d07-8422-f47cd77e116d","Type":"ContainerDied","Data":"f433939d8954878b9fa441023c857823de824e12ef44b8c3756017b09819dc90"} Nov 03 21:38:10.423205 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.423203 4493 scope.go:122] "RemoveContainer" containerID="c50f483e0ace592ceff8af3e9c80add5677055efb6b4550aace8c7fc89271bd8" Nov 03 21:38:10.424569 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.424551 4493 generic.go:356] "Generic (PLEG): container finished" podID="01cddd9d-7cdf-4e9f-8828-525272f3b558" containerID="41c884f0370d9860c2235f4ee60f1e2238f7d08874b4cae716a3ce1f84a4a0d2" exitCode=0 Nov 03 21:38:10.424684 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.424591 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq" event={"ID":"01cddd9d-7cdf-4e9f-8828-525272f3b558","Type":"ContainerDied","Data":"41c884f0370d9860c2235f4ee60f1e2238f7d08874b4cae716a3ce1f84a4a0d2"} Nov 03 21:38:10.424684 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.424646 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq" event={"ID":"01cddd9d-7cdf-4e9f-8828-525272f3b558","Type":"ContainerDied","Data":"f316e028b263640cddfb1a691111f3e3819cd81547e7ba1e757ffac1ed54939e"} Nov 03 21:38:10.424684 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.424604 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq" Nov 03 21:38:10.433775 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.433758 4493 scope.go:122] "RemoveContainer" containerID="c50f483e0ace592ceff8af3e9c80add5677055efb6b4550aace8c7fc89271bd8" Nov 03 21:38:10.434063 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:38:10.434036 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c50f483e0ace592ceff8af3e9c80add5677055efb6b4550aace8c7fc89271bd8\": container with ID starting with c50f483e0ace592ceff8af3e9c80add5677055efb6b4550aace8c7fc89271bd8 not found: ID does not exist" containerID="c50f483e0ace592ceff8af3e9c80add5677055efb6b4550aace8c7fc89271bd8" Nov 03 21:38:10.434135 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.434074 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c50f483e0ace592ceff8af3e9c80add5677055efb6b4550aace8c7fc89271bd8"} err="failed to get container status \"c50f483e0ace592ceff8af3e9c80add5677055efb6b4550aace8c7fc89271bd8\": rpc error: code = NotFound desc = could not find container \"c50f483e0ace592ceff8af3e9c80add5677055efb6b4550aace8c7fc89271bd8\": container with ID starting with c50f483e0ace592ceff8af3e9c80add5677055efb6b4550aace8c7fc89271bd8 not found: ID does not exist" Nov 03 21:38:10.434135 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.434096 4493 scope.go:122] "RemoveContainer" containerID="41c884f0370d9860c2235f4ee60f1e2238f7d08874b4cae716a3ce1f84a4a0d2" Nov 03 21:38:10.442221 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.442204 4493 scope.go:122] "RemoveContainer" containerID="41c884f0370d9860c2235f4ee60f1e2238f7d08874b4cae716a3ce1f84a4a0d2" Nov 03 21:38:10.442491 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:38:10.442472 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41c884f0370d9860c2235f4ee60f1e2238f7d08874b4cae716a3ce1f84a4a0d2\": container with ID starting with 41c884f0370d9860c2235f4ee60f1e2238f7d08874b4cae716a3ce1f84a4a0d2 not found: ID does not exist" containerID="41c884f0370d9860c2235f4ee60f1e2238f7d08874b4cae716a3ce1f84a4a0d2" Nov 03 21:38:10.442565 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.442497 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41c884f0370d9860c2235f4ee60f1e2238f7d08874b4cae716a3ce1f84a4a0d2"} err="failed to get container status \"41c884f0370d9860c2235f4ee60f1e2238f7d08874b4cae716a3ce1f84a4a0d2\": rpc error: code = NotFound desc = could not find container \"41c884f0370d9860c2235f4ee60f1e2238f7d08874b4cae716a3ce1f84a4a0d2\": container with ID starting with 41c884f0370d9860c2235f4ee60f1e2238f7d08874b4cae716a3ce1f84a4a0d2 not found: ID does not exist" Nov 03 21:38:10.593749 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.593646 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc"] Nov 03 21:38:10.645224 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.645193 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-1850fc-6c5c559445-sxzmc"] Nov 03 21:38:10.710817 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.710787 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq"] Nov 03 21:38:10.716150 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:10.716123 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-1850fc-dcbc45895-jx9rq"] Nov 03 21:38:12.559720 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:12.559681 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01cddd9d-7cdf-4e9f-8828-525272f3b558" path="/var/lib/kubelet/pods/01cddd9d-7cdf-4e9f-8828-525272f3b558/volumes" Nov 03 21:38:12.560246 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:12.560226 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec32775d-1e45-4d07-8422-f47cd77e116d" path="/var/lib/kubelet/pods/ec32775d-1e45-4d07-8422-f47cd77e116d/volumes" Nov 03 21:38:13.519804 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:13.519768 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h"] Nov 03 21:38:13.520038 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:13.520009 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h" podUID="e40f1881-b555-4ea8-84a9-cb323a0a106a" containerName="maas-api" containerID="cri-o://24b5ad914dfe4305119c885f9eb7f896ed61adbe877295039e1593aba530b343" gracePeriod=30 Nov 03 21:38:13.777427 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:13.777367 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h" Nov 03 21:38:13.934906 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:13.934877 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e40f1881-b555-4ea8-84a9-cb323a0a106a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e40f1881-b555-4ea8-84a9-cb323a0a106a-metrics-certs\") pod \"e40f1881-b555-4ea8-84a9-cb323a0a106a\" (UID: \"e40f1881-b555-4ea8-84a9-cb323a0a106a\") " Nov 03 21:38:13.935109 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:13.934937 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e40f1881-b555-4ea8-84a9-cb323a0a106a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e40f1881-b555-4ea8-84a9-cb323a0a106a-openshift-service-ca\") pod \"e40f1881-b555-4ea8-84a9-cb323a0a106a\" (UID: \"e40f1881-b555-4ea8-84a9-cb323a0a106a\") " Nov 03 21:38:13.935109 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:13.934997 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e40f1881-b555-4ea8-84a9-cb323a0a106a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e40f1881-b555-4ea8-84a9-cb323a0a106a-maas-api-tls\") pod \"e40f1881-b555-4ea8-84a9-cb323a0a106a\" (UID: \"e40f1881-b555-4ea8-84a9-cb323a0a106a\") " Nov 03 21:38:13.935109 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:13.935070 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e40f1881-b555-4ea8-84a9-cb323a0a106a-kube-api-access-z5dj4\" (UniqueName: \"kubernetes.io/projected/e40f1881-b555-4ea8-84a9-cb323a0a106a-kube-api-access-z5dj4\") pod \"e40f1881-b555-4ea8-84a9-cb323a0a106a\" (UID: \"e40f1881-b555-4ea8-84a9-cb323a0a106a\") " Nov 03 21:38:13.935347 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:13.935301 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e40f1881-b555-4ea8-84a9-cb323a0a106a-openshift-service-ca" pod "e40f1881-b555-4ea8-84a9-cb323a0a106a" (UID: "e40f1881-b555-4ea8-84a9-cb323a0a106a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:38:13.935473 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:13.935456 4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e40f1881-b555-4ea8-84a9-cb323a0a106a-openshift-service-ca\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:13.937248 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:13.937223 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e40f1881-b555-4ea8-84a9-cb323a0a106a-metrics-certs" pod "e40f1881-b555-4ea8-84a9-cb323a0a106a" (UID: "e40f1881-b555-4ea8-84a9-cb323a0a106a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:38:13.937435 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:13.937262 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e40f1881-b555-4ea8-84a9-cb323a0a106a-maas-api-tls" pod "e40f1881-b555-4ea8-84a9-cb323a0a106a" (UID: "e40f1881-b555-4ea8-84a9-cb323a0a106a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:38:13.937435 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:13.937301 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e40f1881-b555-4ea8-84a9-cb323a0a106a-kube-api-access-z5dj4" pod "e40f1881-b555-4ea8-84a9-cb323a0a106a" (UID: "e40f1881-b555-4ea8-84a9-cb323a0a106a"). InnerVolumeSpecName "kube-api-access-z5dj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:38:14.037232 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:14.037141 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5dj4\" (UniqueName: \"kubernetes.io/projected/e40f1881-b555-4ea8-84a9-cb323a0a106a-kube-api-access-z5dj4\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:14.037232 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:14.037184 4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e40f1881-b555-4ea8-84a9-cb323a0a106a-metrics-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:14.037232 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:14.037202 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e40f1881-b555-4ea8-84a9-cb323a0a106a-maas-api-tls\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:14.446807 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:14.446776 4493 generic.go:356] "Generic (PLEG): container finished" podID="e40f1881-b555-4ea8-84a9-cb323a0a106a" containerID="24b5ad914dfe4305119c885f9eb7f896ed61adbe877295039e1593aba530b343" exitCode=0 Nov 03 21:38:14.446984 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:14.446833 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h" Nov 03 21:38:14.446984 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:14.446846 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h" event={"ID":"e40f1881-b555-4ea8-84a9-cb323a0a106a","Type":"ContainerDied","Data":"24b5ad914dfe4305119c885f9eb7f896ed61adbe877295039e1593aba530b343"} Nov 03 21:38:14.446984 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:14.446888 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h" event={"ID":"e40f1881-b555-4ea8-84a9-cb323a0a106a","Type":"ContainerDied","Data":"d6ccfafc809b840c3ff57ffdb88d7b8aaaefffc6c8031597fb2d3aefde395fd5"} Nov 03 21:38:14.446984 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:14.446909 4493 scope.go:122] "RemoveContainer" containerID="24b5ad914dfe4305119c885f9eb7f896ed61adbe877295039e1593aba530b343" Nov 03 21:38:14.456770 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:14.456749 4493 scope.go:122] "RemoveContainer" containerID="24b5ad914dfe4305119c885f9eb7f896ed61adbe877295039e1593aba530b343" Nov 03 21:38:14.457042 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:38:14.457022 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24b5ad914dfe4305119c885f9eb7f896ed61adbe877295039e1593aba530b343\": container with ID starting with 24b5ad914dfe4305119c885f9eb7f896ed61adbe877295039e1593aba530b343 not found: ID does not exist" containerID="24b5ad914dfe4305119c885f9eb7f896ed61adbe877295039e1593aba530b343" Nov 03 21:38:14.457104 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:14.457050 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24b5ad914dfe4305119c885f9eb7f896ed61adbe877295039e1593aba530b343"} err="failed to get container status \"24b5ad914dfe4305119c885f9eb7f896ed61adbe877295039e1593aba530b343\": rpc error: code = NotFound desc = could not find container \"24b5ad914dfe4305119c885f9eb7f896ed61adbe877295039e1593aba530b343\": container with ID starting with 24b5ad914dfe4305119c885f9eb7f896ed61adbe877295039e1593aba530b343 not found: ID does not exist" Nov 03 21:38:14.615193 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:14.615161 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h"] Nov 03 21:38:14.666970 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:14.666940 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1850fc-658dbf68c-49j4h"] Nov 03 21:38:16.560092 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:16.560059 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e40f1881-b555-4ea8-84a9-cb323a0a106a" path="/var/lib/kubelet/pods/e40f1881-b555-4ea8-84a9-cb323a0a106a/volumes" Nov 03 21:38:21.744112 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:38:21.744075 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:38:21.863273 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:21.863233 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2"] Nov 03 21:38:21.863579 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:21.863552 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" podUID="c35a8aed-f40e-48dd-9138-19f5ce831a33" containerName="istio-proxy" containerID="cri-o://f8175a7038b4c22ebcbc6b9b81656701791ca87f68bbbb6e998f0e879ab18901" gracePeriod=30 Nov 03 21:38:27.123594 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.123572 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:38:27.168227 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.168157 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-data\") pod \"c35a8aed-f40e-48dd-9138-19f5ce831a33\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " Nov 03 21:38:27.168227 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.168197 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-token\" (UniqueName: \"kubernetes.io/projected/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-token\") pod \"c35a8aed-f40e-48dd-9138-19f5ce831a33\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " Nov 03 21:38:27.168227 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.168215 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c35a8aed-f40e-48dd-9138-19f5ce831a33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c35a8aed-f40e-48dd-9138-19f5ce831a33-istiod-ca-cert\") pod \"c35a8aed-f40e-48dd-9138-19f5ce831a33\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " Nov 03 21:38:27.168531 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.168247 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-podinfo\") pod \"c35a8aed-f40e-48dd-9138-19f5ce831a33\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " Nov 03 21:38:27.168531 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.168307 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-workload-socket\") pod \"c35a8aed-f40e-48dd-9138-19f5ce831a33\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " Nov 03 21:38:27.168531 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.168372 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c35a8aed-f40e-48dd-9138-19f5ce831a33-kube-api-access-j9vcv\" (UniqueName: \"kubernetes.io/projected/c35a8aed-f40e-48dd-9138-19f5ce831a33-kube-api-access-j9vcv\") pod \"c35a8aed-f40e-48dd-9138-19f5ce831a33\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " Nov 03 21:38:27.168531 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.168420 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-credential-socket\") pod \"c35a8aed-f40e-48dd-9138-19f5ce831a33\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " Nov 03 21:38:27.168531 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.168474 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-data" pod "c35a8aed-f40e-48dd-9138-19f5ce831a33" (UID: "c35a8aed-f40e-48dd-9138-19f5ce831a33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:38:27.168531 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.168502 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-workload-certs\") pod \"c35a8aed-f40e-48dd-9138-19f5ce831a33\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " Nov 03 21:38:27.168856 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.168537 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-envoy\") pod \"c35a8aed-f40e-48dd-9138-19f5ce831a33\" (UID: \"c35a8aed-f40e-48dd-9138-19f5ce831a33\") " Nov 03 21:38:27.168856 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.168726 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-credential-socket" pod "c35a8aed-f40e-48dd-9138-19f5ce831a33" (UID: "c35a8aed-f40e-48dd-9138-19f5ce831a33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:38:27.168856 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.168778 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-workload-socket" pod "c35a8aed-f40e-48dd-9138-19f5ce831a33" (UID: "c35a8aed-f40e-48dd-9138-19f5ce831a33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:38:27.168856 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.168832 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-workload-certs" pod "c35a8aed-f40e-48dd-9138-19f5ce831a33" (UID: "c35a8aed-f40e-48dd-9138-19f5ce831a33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:38:27.169088 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.169051 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c35a8aed-f40e-48dd-9138-19f5ce831a33-istiod-ca-cert" pod "c35a8aed-f40e-48dd-9138-19f5ce831a33" (UID: "c35a8aed-f40e-48dd-9138-19f5ce831a33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:38:27.169618 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.169592 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-workload-certs\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:27.169759 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.169621 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-data\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:27.169759 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.169637 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c35a8aed-f40e-48dd-9138-19f5ce831a33-istiod-ca-cert\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:27.169759 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.169653 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-workload-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:27.169759 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.169668 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-credential-socket\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:27.170803 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.170775 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-token" pod "c35a8aed-f40e-48dd-9138-19f5ce831a33" (UID: "c35a8aed-f40e-48dd-9138-19f5ce831a33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:38:27.170908 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.170889 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35a8aed-f40e-48dd-9138-19f5ce831a33-kube-api-access-j9vcv" pod "c35a8aed-f40e-48dd-9138-19f5ce831a33" (UID: "c35a8aed-f40e-48dd-9138-19f5ce831a33"). InnerVolumeSpecName "kube-api-access-j9vcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:38:27.171236 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.171219 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-envoy" pod "c35a8aed-f40e-48dd-9138-19f5ce831a33" (UID: "c35a8aed-f40e-48dd-9138-19f5ce831a33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:38:27.171236 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.171224 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-podinfo" pod "c35a8aed-f40e-48dd-9138-19f5ce831a33" (UID: "c35a8aed-f40e-48dd-9138-19f5ce831a33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:38:27.270093 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.270066 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9vcv\" (UniqueName: \"kubernetes.io/projected/c35a8aed-f40e-48dd-9138-19f5ce831a33-kube-api-access-j9vcv\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:27.270093 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.270093 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-envoy\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:27.270278 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.270102 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-token\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:27.270278 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.270111 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c35a8aed-f40e-48dd-9138-19f5ce831a33-istio-podinfo\") on node \"ip-10-0-138-26.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:27.511510 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.511412 4493 generic.go:356] "Generic (PLEG): container finished" podID="c35a8aed-f40e-48dd-9138-19f5ce831a33" containerID="f8175a7038b4c22ebcbc6b9b81656701791ca87f68bbbb6e998f0e879ab18901" exitCode=0 Nov 03 21:38:27.511510 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.511448 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" event={"ID":"c35a8aed-f40e-48dd-9138-19f5ce831a33","Type":"ContainerDied","Data":"f8175a7038b4c22ebcbc6b9b81656701791ca87f68bbbb6e998f0e879ab18901"} Nov 03 21:38:27.511510 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.511480 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" Nov 03 21:38:27.511889 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.511515 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2" event={"ID":"c35a8aed-f40e-48dd-9138-19f5ce831a33","Type":"ContainerDied","Data":"6b6f76962e56987708516dcdfcd072efbacddfc8d6020f111fb5eeb01cbb2203"} Nov 03 21:38:27.511889 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.511538 4493 scope.go:122] "RemoveContainer" containerID="f8175a7038b4c22ebcbc6b9b81656701791ca87f68bbbb6e998f0e879ab18901" Nov 03 21:38:27.521393 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.521373 4493 scope.go:122] "RemoveContainer" containerID="556376d767803c302785632b97a2141dfb0e3727e6ebd9968909985a771295f1" Nov 03 21:38:27.529916 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.529901 4493 scope.go:122] "RemoveContainer" containerID="f8175a7038b4c22ebcbc6b9b81656701791ca87f68bbbb6e998f0e879ab18901" Nov 03 21:38:27.530181 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:38:27.530163 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8175a7038b4c22ebcbc6b9b81656701791ca87f68bbbb6e998f0e879ab18901\": container with ID starting with f8175a7038b4c22ebcbc6b9b81656701791ca87f68bbbb6e998f0e879ab18901 not found: ID does not exist" containerID="f8175a7038b4c22ebcbc6b9b81656701791ca87f68bbbb6e998f0e879ab18901" Nov 03 21:38:27.530229 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.530188 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8175a7038b4c22ebcbc6b9b81656701791ca87f68bbbb6e998f0e879ab18901"} err="failed to get container status \"f8175a7038b4c22ebcbc6b9b81656701791ca87f68bbbb6e998f0e879ab18901\": rpc error: code = NotFound desc = could not find container \"f8175a7038b4c22ebcbc6b9b81656701791ca87f68bbbb6e998f0e879ab18901\": container with ID starting with f8175a7038b4c22ebcbc6b9b81656701791ca87f68bbbb6e998f0e879ab18901 not found: ID does not exist" Nov 03 21:38:27.530229 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.530206 4493 scope.go:122] "RemoveContainer" containerID="556376d767803c302785632b97a2141dfb0e3727e6ebd9968909985a771295f1" Nov 03 21:38:27.530445 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:38:27.530429 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"556376d767803c302785632b97a2141dfb0e3727e6ebd9968909985a771295f1\": container with ID starting with 556376d767803c302785632b97a2141dfb0e3727e6ebd9968909985a771295f1 not found: ID does not exist" containerID="556376d767803c302785632b97a2141dfb0e3727e6ebd9968909985a771295f1" Nov 03 21:38:27.530492 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.530452 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"556376d767803c302785632b97a2141dfb0e3727e6ebd9968909985a771295f1"} err="failed to get container status \"556376d767803c302785632b97a2141dfb0e3727e6ebd9968909985a771295f1\": rpc error: code = NotFound desc = could not find container \"556376d767803c302785632b97a2141dfb0e3727e6ebd9968909985a771295f1\": container with ID starting with 556376d767803c302785632b97a2141dfb0e3727e6ebd9968909985a771295f1 not found: ID does not exist" Nov 03 21:38:27.680235 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.680206 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2"] Nov 03 21:38:27.731710 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:27.731679 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-1850fc-openshift-default-577d964db5-qk6c2"] Nov 03 21:38:28.560443 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:28.560371 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c35a8aed-f40e-48dd-9138-19f5ce831a33" path="/var/lib/kubelet/pods/c35a8aed-f40e-48dd-9138-19f5ce831a33/volumes" Nov 03 21:38:43.154216 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:43.154172 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-65lk6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:38:43.154937 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:43.154902 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-65lk6" podUID="b3601ef6-138a-4d85-8bd1-3894244e6496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:38:43.652580 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:43.652540 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-cqjpw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:38:43.653571 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:43.653533 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-cqjpw" podUID="ecce3865-c792-4c76-ba4b-58c563321e8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:38:49.328214 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:49.328182 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 21:38:49.415154 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:38:49.415125 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82vml_d48c0b65-db57-43b2-8e7c-6e9e8feddf89/ovn-acl-logging/0.log" Nov 03 21:38:51.743585 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:38:51.743546 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:39:21.743747 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:39:21.743707 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:39:51.743742 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:39:51.743703 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:39:54.056305 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:39:54.056248 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-49ddb_07dea7e4-f5d0-485f-ad61-28b3ac0d9f85/manager/0.log" Nov 03 21:40:06.421919 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:40:06.421885 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-59fffb9f98-4zvg6_dfe60236-83d1-4419-85c2-b99031445c6b/maas-api/0.log" Nov 03 21:40:09.810800 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:40:09.810772 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-5k9hh_eb2d5dbe-0f4d-4b1e-88b9-913c6df7d20a/kuadrant-console-plugin/0.log" Nov 03 21:40:12.018168 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:40:12.018133 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-bb87d7467-shtb7_59c188bf-2663-4d7d-8b1a-cd3320f45b28/kube-auth-proxy/0.log" Nov 03 21:40:12.235909 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:40:12.235883 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-65lk6_b3601ef6-138a-4d85-8bd1-3894244e6496/istio-proxy/0.log" Nov 03 21:40:12.486082 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:40:12.486050 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-cqjpw_ecce3865-c792-4c76-ba4b-58c563321e8a/istio-proxy/0.log" Nov 03 21:40:14.037318 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:40:14.037290 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-xd49n_197b41b5-4233-40c8-855a-38b91aedf4eb/istio-proxy/0.log" Nov 03 21:40:14.295903 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:40:14.295880 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-zgpq2_da127cda-8ef4-43c8-8eb8-2fe89b52ffcd/istio-proxy/0.log" Nov 03 21:40:21.744222 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:40:21.744188 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:40:26.263484 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:40:26.263445 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-bw856_32fb3160-1d88-4cf6-ac03-4e2f030414cc/main/0.log" Nov 03 21:40:27.019443 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:40:27.019411 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-vkr64_8e855d53-7358-4981-8313-16b0dbf17475/main/0.log" Nov 03 21:40:27.528437 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:40:27.528406 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-4bmw5_8e6fd928-16a6-4624-b8a1-8e3027433637/main/0.log" Nov 03 21:40:38.057749 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:40:38.057719 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-49ddb_07dea7e4-f5d0-485f-ad61-28b3ac0d9f85/manager/0.log" Nov 03 21:40:39.917289 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:40:39.917252 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-59fffb9f98-4zvg6_dfe60236-83d1-4419-85c2-b99031445c6b/maas-api/0.log" Nov 03 21:40:42.944244 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:40:42.944211 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-5k9hh_eb2d5dbe-0f4d-4b1e-88b9-913c6df7d20a/kuadrant-console-plugin/0.log" Nov 03 21:40:44.810531 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:40:44.810500 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-bb87d7467-shtb7_59c188bf-2663-4d7d-8b1a-cd3320f45b28/kube-auth-proxy/0.log" Nov 03 21:40:45.022253 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:40:45.022222 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-65lk6_b3601ef6-138a-4d85-8bd1-3894244e6496/istio-proxy/0.log" Nov 03 21:40:45.268288 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:40:45.268214 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-cqjpw_ecce3865-c792-4c76-ba4b-58c563321e8a/istio-proxy/0.log" Nov 03 21:40:46.798251 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:40:46.798217 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-xd49n_197b41b5-4233-40c8-855a-38b91aedf4eb/istio-proxy/0.log" Nov 03 21:40:47.052211 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:40:47.052131 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-zgpq2_da127cda-8ef4-43c8-8eb8-2fe89b52ffcd/istio-proxy/0.log" Nov 03 21:40:48.458986 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:40:48.458955 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-bw856_32fb3160-1d88-4cf6-ac03-4e2f030414cc/main/0.log" Nov 03 21:40:49.164066 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:40:49.164036 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-vkr64_8e855d53-7358-4981-8313-16b0dbf17475/main/0.log" Nov 03 21:40:49.643179 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:40:49.643157 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-4bmw5_8e6fd928-16a6-4624-b8a1-8e3027433637/main/0.log" Nov 03 21:40:51.743428 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:40:51.743385 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:41:21.743137 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:41:21.743099 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:41:51.744053 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:41:51.743968 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:42:21.743931 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:42:21.743894 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:42:32.442015 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:32.441982 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2sjv9_ef30d813-b7ed-417d-a9fe-85118bf77a95/global-pull-secret-syncer/0.log" Nov 03 21:42:35.835900 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:35.835871 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ccxwr_e99c666f-7bb3-4856-a2fc-935ee184dae3/konnectivity-agent/0.log" Nov 03 21:42:37.587560 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:37.587529 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-26.us-west-2.compute.internal_733c8a62db18973abc6d8fb96bb89db1/haproxy/0.log" Nov 03 21:42:41.948826 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:41.948795 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dd722/perf-node-gather-daemonset-nnsg9"] Nov 03 21:42:41.949245 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:41.949197 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e40f1881-b555-4ea8-84a9-cb323a0a106a" containerName="maas-api" Nov 03 21:42:41.949245 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:41.949209 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e40f1881-b555-4ea8-84a9-cb323a0a106a" containerName="maas-api" Nov 03 21:42:41.949245 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:41.949225 4493 cpu_manager.go:414] "containerMap: removing container" podUID="c35a8aed-f40e-48dd-9138-19f5ce831a33" containerName="istio-proxy" Nov 03 21:42:41.949245 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:41.949230 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c35a8aed-f40e-48dd-9138-19f5ce831a33" containerName="istio-proxy" Nov 03 21:42:41.949245 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:41.949240 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ec32775d-1e45-4d07-8422-f47cd77e116d" containerName="payload-processing" Nov 03 21:42:41.949245 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:41.949244 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec32775d-1e45-4d07-8422-f47cd77e116d" containerName="payload-processing" Nov 03 21:42:41.949484 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:41.949256 4493 cpu_manager.go:414] "containerMap: removing container" podUID="01cddd9d-7cdf-4e9f-8828-525272f3b558" containerName="payload-pre-processing" Nov 03 21:42:41.949484 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:41.949261 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01cddd9d-7cdf-4e9f-8828-525272f3b558" containerName="payload-pre-processing" Nov 03 21:42:41.949484 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:41.949346 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c35a8aed-f40e-48dd-9138-19f5ce831a33" containerName="istio-proxy" Nov 03 21:42:41.949484 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:41.949355 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec32775d-1e45-4d07-8422-f47cd77e116d" containerName="payload-processing" Nov 03 21:42:41.949484 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:41.949365 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="01cddd9d-7cdf-4e9f-8828-525272f3b558" containerName="payload-pre-processing" Nov 03 21:42:41.949484 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:41.949380 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e40f1881-b555-4ea8-84a9-cb323a0a106a" containerName="maas-api" Nov 03 21:42:41.952615 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:41.952598 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dd722/perf-node-gather-daemonset-nnsg9" Nov 03 21:42:42.006158 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:42.006134 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dd722\"/\"openshift-service-ca.crt\"" Nov 03 21:42:42.027257 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:42.027233 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hptxg\" (UniqueName: \"kubernetes.io/projected/6049cb15-f8dd-49dc-84cd-b6d7a8dde371-kube-api-access-hptxg\") pod \"perf-node-gather-daemonset-nnsg9\" (UID: \"6049cb15-f8dd-49dc-84cd-b6d7a8dde371\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-nnsg9" Nov 03 21:42:42.027358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:42.027268 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6049cb15-f8dd-49dc-84cd-b6d7a8dde371-lib-modules\") pod \"perf-node-gather-daemonset-nnsg9\" (UID: \"6049cb15-f8dd-49dc-84cd-b6d7a8dde371\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-nnsg9" Nov 03 21:42:42.027358 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:42.027294 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6049cb15-f8dd-49dc-84cd-b6d7a8dde371-sys\") pod \"perf-node-gather-daemonset-nnsg9\" (UID: \"6049cb15-f8dd-49dc-84cd-b6d7a8dde371\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-nnsg9" Nov 03 21:42:42.027502 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:42.027488 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6049cb15-f8dd-49dc-84cd-b6d7a8dde371-proc\") pod \"perf-node-gather-daemonset-nnsg9\" (UID: \"6049cb15-f8dd-49dc-84cd-b6d7a8dde371\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-nnsg9" Nov 03 21:42:42.027541 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:42.027523 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6049cb15-f8dd-49dc-84cd-b6d7a8dde371-podres\") pod \"perf-node-gather-daemonset-nnsg9\" (UID: \"6049cb15-f8dd-49dc-84cd-b6d7a8dde371\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-nnsg9" Nov 03 21:42:42.058286 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:42.058262 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dd722\"/\"default-dockercfg-85g9m\"" Nov 03 21:42:42.058431 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:42.058312 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dd722\"/\"kube-root-ca.crt\"" Nov 03 21:42:42.060227 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:42.060207 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dd722/perf-node-gather-daemonset-nnsg9"] Nov 03 21:42:42.128742 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:42.128711 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6049cb15-f8dd-49dc-84cd-b6d7a8dde371-proc\") pod \"perf-node-gather-daemonset-nnsg9\" (UID: \"6049cb15-f8dd-49dc-84cd-b6d7a8dde371\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-nnsg9" Nov 03 21:42:42.128918 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:42.128752 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6049cb15-f8dd-49dc-84cd-b6d7a8dde371-podres\") pod \"perf-node-gather-daemonset-nnsg9\" (UID: \"6049cb15-f8dd-49dc-84cd-b6d7a8dde371\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-nnsg9" Nov 03 21:42:42.128918 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:42.128776 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hptxg\" (UniqueName: \"kubernetes.io/projected/6049cb15-f8dd-49dc-84cd-b6d7a8dde371-kube-api-access-hptxg\") pod \"perf-node-gather-daemonset-nnsg9\" (UID: \"6049cb15-f8dd-49dc-84cd-b6d7a8dde371\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-nnsg9" Nov 03 21:42:42.128918 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:42.128798 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6049cb15-f8dd-49dc-84cd-b6d7a8dde371-lib-modules\") pod \"perf-node-gather-daemonset-nnsg9\" (UID: \"6049cb15-f8dd-49dc-84cd-b6d7a8dde371\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-nnsg9" Nov 03 21:42:42.128918 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:42.128829 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6049cb15-f8dd-49dc-84cd-b6d7a8dde371-sys\") pod \"perf-node-gather-daemonset-nnsg9\" (UID: \"6049cb15-f8dd-49dc-84cd-b6d7a8dde371\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-nnsg9" Nov 03 21:42:42.128918 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:42.128841 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6049cb15-f8dd-49dc-84cd-b6d7a8dde371-proc\") pod \"perf-node-gather-daemonset-nnsg9\" (UID: \"6049cb15-f8dd-49dc-84cd-b6d7a8dde371\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-nnsg9" Nov 03 21:42:42.128918 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:42.128851 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6049cb15-f8dd-49dc-84cd-b6d7a8dde371-podres\") pod \"perf-node-gather-daemonset-nnsg9\" (UID: \"6049cb15-f8dd-49dc-84cd-b6d7a8dde371\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-nnsg9" Nov 03 21:42:42.128918 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:42.128890 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6049cb15-f8dd-49dc-84cd-b6d7a8dde371-sys\") pod \"perf-node-gather-daemonset-nnsg9\" (UID: \"6049cb15-f8dd-49dc-84cd-b6d7a8dde371\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-nnsg9" Nov 03 21:42:42.128918 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:42.128923 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6049cb15-f8dd-49dc-84cd-b6d7a8dde371-lib-modules\") pod \"perf-node-gather-daemonset-nnsg9\" (UID: \"6049cb15-f8dd-49dc-84cd-b6d7a8dde371\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-nnsg9" Nov 03 21:42:42.186936 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:42.186909 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hptxg\" (UniqueName: \"kubernetes.io/projected/6049cb15-f8dd-49dc-84cd-b6d7a8dde371-kube-api-access-hptxg\") pod \"perf-node-gather-daemonset-nnsg9\" (UID: \"6049cb15-f8dd-49dc-84cd-b6d7a8dde371\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-nnsg9" Nov 03 21:42:42.265847 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:42.265774 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dd722/perf-node-gather-daemonset-nnsg9" Nov 03 21:42:42.638372 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:42.638346 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dd722/perf-node-gather-daemonset-nnsg9"] Nov 03 21:42:42.639626 ip-10-0-138-26 kubenswrapper[4493]: W1103 21:42:42.639594 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6049cb15_f8dd_49dc_84cd_b6d7a8dde371.slice/crio-e694815d74066f5cd9ff172aad2958950870ecf17139a1f9478fffa67d9b2103 WatchSource:0}: Error finding container e694815d74066f5cd9ff172aad2958950870ecf17139a1f9478fffa67d9b2103: Status 404 returned error can't find the container with id e694815d74066f5cd9ff172aad2958950870ecf17139a1f9478fffa67d9b2103 Nov 03 21:42:42.720066 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:42.720044 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-5k9hh_eb2d5dbe-0f4d-4b1e-88b9-913c6df7d20a/kuadrant-console-plugin/0.log" Nov 03 21:42:42.818289 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:42.818261 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dd722/perf-node-gather-daemonset-nnsg9" event={"ID":"6049cb15-f8dd-49dc-84cd-b6d7a8dde371","Type":"ContainerStarted","Data":"04da14553cf9f819380c7f912cf703fd7918a70bdb5347eecefe47cd370db11c"} Nov 03 21:42:42.818456 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:42.818305 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dd722/perf-node-gather-daemonset-nnsg9" event={"ID":"6049cb15-f8dd-49dc-84cd-b6d7a8dde371","Type":"ContainerStarted","Data":"e694815d74066f5cd9ff172aad2958950870ecf17139a1f9478fffa67d9b2103"} Nov 03 21:42:42.818456 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:42.818395 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dd722/perf-node-gather-daemonset-nnsg9" Nov 03 21:42:42.935706 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:42.935600 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dd722/perf-node-gather-daemonset-nnsg9" podStartSLOduration=1.9355880779999999 podStartE2EDuration="1.935588078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:42.934534956 +0000 UTC m=+4434.492525455" watchObservedRunningTime="2026-11-03 21:42:42.935588078 +0000 UTC m=+4434.493578576" Nov 03 21:42:48.833120 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:48.833094 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dd722/perf-node-gather-daemonset-nnsg9" Nov 03 21:42:49.863933 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:49.863905 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-rlrxx_76800d8f-c9c9-45f3-8a2c-aa8f1c914026/kube-state-metrics/0.log" Nov 03 21:42:50.261089 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:50.261014 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-rlrxx_76800d8f-c9c9-45f3-8a2c-aa8f1c914026/kube-rbac-proxy-main/0.log" Nov 03 21:42:50.663398 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:50.663368 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-rlrxx_76800d8f-c9c9-45f3-8a2c-aa8f1c914026/kube-rbac-proxy-self/0.log" Nov 03 21:42:51.743444 ip-10-0-138-26 kubenswrapper[4493]: E1103 21:42:51.743407 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-67nxb" containerName="extractor" Nov 03 21:42:54.336060 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:54.336025 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vjzhq_f779b3c3-bf33-44c8-8de5-cd75cece8e82/node-exporter/0.log" Nov 03 21:42:54.737828 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:54.737756 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vjzhq_f779b3c3-bf33-44c8-8de5-cd75cece8e82/kube-rbac-proxy/0.log" Nov 03 21:42:55.138773 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:55.138744 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vjzhq_f779b3c3-bf33-44c8-8de5-cd75cece8e82/init-textfile/0.log" Nov 03 21:42:55.544912 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:55.544830 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-5ckdt_09f023f0-38ef-4d16-a5d7-f3e309d853ba/kube-rbac-proxy-main/0.log" Nov 03 21:42:55.947909 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:55.947883 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-5ckdt_09f023f0-38ef-4d16-a5d7-f3e309d853ba/kube-rbac-proxy-self/0.log" Nov 03 21:42:56.349946 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:56.349917 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-5ckdt_09f023f0-38ef-4d16-a5d7-f3e309d853ba/openshift-state-metrics/0.log" Nov 03 21:42:56.901829 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:56.901795 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5a03d57d-47f6-4ae1-9e1a-62cbe72d453b/prometheus/0.log" Nov 03 21:42:57.304270 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:57.304241 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5a03d57d-47f6-4ae1-9e1a-62cbe72d453b/config-reloader/0.log" Nov 03 21:42:57.707389 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:57.707278 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5a03d57d-47f6-4ae1-9e1a-62cbe72d453b/thanos-sidecar/0.log" Nov 03 21:42:58.108718 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:58.108687 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5a03d57d-47f6-4ae1-9e1a-62cbe72d453b/kube-rbac-proxy-web/0.log" Nov 03 21:42:58.512863 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:58.512789 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5a03d57d-47f6-4ae1-9e1a-62cbe72d453b/kube-rbac-proxy/0.log" Nov 03 21:42:58.916218 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:58.916189 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5a03d57d-47f6-4ae1-9e1a-62cbe72d453b/kube-rbac-proxy-thanos/0.log" Nov 03 21:42:59.321088 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:59.321054 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5a03d57d-47f6-4ae1-9e1a-62cbe72d453b/init-config-reloader/0.log" Nov 03 21:42:59.733513 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:42:59.733427 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-bbqpt_1fb24a85-7171-4924-9578-fe5e5fe58ca5/prometheus-operator/0.log" Nov 03 21:43:00.129084 ip-10-0-138-26 kubenswrapper[4493]: I1103 21:43:00.129049 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-bbqpt_1fb24a85-7171-4924-9578-fe5e5fe58ca5/kube-rbac-proxy/0.log"